Reusing a verified answer in chat
When the same question returns in another language, a verified earlier answer matches thinking mode at about a fifth of its tokens, once a meaning check stops it reaching the wrong question.
Updated
Measured on maths questions with two model versions. A research result: not yet used for answers in the chat service.
In this paper 11 sections
The question: should a model solve the same problem twice?
People ask the same things. A maths problem, a tax rule or an exam date reaches a chat service many times, often in different languages. Today the model works each one out from scratch, and when thinking mode is on it spends hundreds of extra tokens doing so. NIF's question for the general model is narrow: when a question has already been answered and that answer was verified, can the next asker get it reused, faster and at least as accurately?
How an answer becomes a recipe
The first time a question appears, the model answers it twice: once directly and once with thinking. The answer is kept as a recipe only if both give the same final number. No answer key is consulted, as in real use, where none exists. Agreement is a weak check, and the results below say how weak: 17 of 228 recipes that passed it were still wrong, because two answers can agree on the same wrong number.
How a recipe is offered
When a later question arrives, a recipe is offered only if the new question is the same problem. The first test is that its numbers match exactly, which works across languages because the translations keep the same digits. The second test, added after the first runs, is a multilingual relevance model reading both questions and scoring whether they mean the same thing. A matching recipe is placed in the user's message with an instruction to check that it applies, never sent as the answer on its own.
Accuracy, three runs
On 349 questions asked again in another language, NIF answered 307, 307 and 304 correctly in three runs, against 288, 288 and 287 for the plain model. On the same questions it fixed 23 to 28 answers the plain model got wrong and broke 6 to 9, a significant gain in every run (exact paired p between 0.001 and 0.003).
| Run | Plain | Thinking | NIF | Tokens: thinking vs NIF | Wrong matches |
|---|---|---|---|---|---|
| v13d, numbers-only matchingNIF vs plain: 26 fixed, 7 broken (p = 0.001). NIF vs thinking: 28 vs 17 (p = 0.14). | 288/349 | 296/349 | 307/349 | 250k vs 46k (23.6 s vs 5.1 s) | 21/508 |
| v12, numbers-only matchingNIF vs plain: 28 fixed, 9 broken (p = 0.003). NIF vs thinking: 26 vs 18 (p = 0.29). | 288/349 | 299/349 | 307/349 | 252k vs 48k (24.2 s vs 5.5 s) | 20/508 |
| v13d, numbers + meaning checkNIF vs plain: 23 fixed, 6 broken (p = 0.002). NIF vs thinking: 19 vs 22 (p = 0.76). | 287/349 | 307/349 | 304/349 | 244k vs 47k (23.6 s vs 5.3 s) | 0/508 |
Thinking's own score moved from 296 to 307 between runs with nothing changed: batching makes the server's greedy answers vary, so differences of a few questions are noise.
View methods and limits
GSM8K-Indic questions that appear more than once in different languages; the first appearance is answered directly and with thinking and becomes a recipe only if both give the same number. Later appearances are answered three ways on identical questions, thinking off unless stated, greedy, on the same server. Significance is an exact paired test on the questions where two setups differ. The meaning check uses a public multilingual reranker; its threshold was calibrated on the first run and then confirmed in a separate full run. Maths only; one suite.
What this does not establishAgainst thinking mode: equal, not better
An earlier version of this note said NIF scored above thinking mode. The later runs do not support that. Thinking scored 296, 299 and 307 on identical questions in three runs, because the server batches requests differently each time and greedy answers shift with it; two runs of the same server without any change differ on about a third of answer texts. Against thinking, NIF's difference is 28 versus 17, 26 versus 18 and 19 versus 22 questions, none of them significant. The honest statement is that NIF matches thinking mode's accuracy here.
What it costs
Matching thinking's accuracy is useful because of the cost. NIF answers used about 46 to 48 thousand tokens in total against 244 to 252 thousand for thinking, roughly a fifth, and arrived in a median 5.1 to 5.5 seconds against 23.6 to 24.2. They also used fewer tokens than the plain model, about two thirds of its 73 to 75 thousand, because a worked solution is already available. One cost is visible in the text: some reused answers are very short, a few words and the number, which is correct but less explanatory than a user may want.
Accuracy (%)
Run 1 · v13dNumbers only
- Plain82.5%
- Thinking84.8%
- NIF88.0%
Run 2 · v12Numbers only
- Plain82.5%
- Thinking85.7%
- NIF88.0%
Run 3 · v13dMeaning check
- Plain82.2%
- Thinking88.0%
- NIF87.1%
Answer tokens (thousands)
Run 1 · v13dNumbers only
- Plain73.4k
- Thinking250.2k
- NIF45.8k
Run 2 · v12Numbers only
- Plain75.2k
- Thinking252.5k
- NIF48.1k
Run 3 · v13dMeaning check
- Plain73.5k
- Thinking244.1k
- NIF47.2k
Median time (seconds)
Run 1 · v13dNumbers only
- Thinking23.6 s
- NIF5.1 s
Run 2 · v12Numbers only
- Thinking24.2 s
- NIF5.5 s
Run 3 · v13dMeaning check
- Thinking23.6 s
- NIF5.3 s
Plain-model median times are not in the three-run artifact and are not inferred.
Saved nif-chat.json, measured 2026-10-20. Answer-token totals are not a full system-cost accounting. Accuracy differences against thinking are not statistically settled; batching also changes greedy answers between runs.
The problem the first runs exposed
Matching by numbers alone was not safe. Different problems sometimes contain the same numbers, and when the model was given a recipe for a different problem it did not notice: on 21 unrelated questions in the first run, accuracy fell from 18 correct to 9. A closer audit found 39 such wrong matches among 329 offers, not 21, because 18 returning questions had also been matched to a different problem that happened to be answered first.
A meaning check, calibrated
A multilingual relevance model already used for retrieval scored each pair: the new question against the question the recipe was verified on. It separated true repeats from wrong matches almost perfectly, with an area under the curve of 0.998. Every wrong match scored 0.051 or lower; the median true repeat scored 0.953. At a threshold of 0.1, about twice the highest wrong score, it kept 270 of 290 true repeats and blocked all 39 wrong matches. The 20 true repeats it drops all involve romanised text, Hindi or Tamil written in Latin script, where the relevance model is weaker; those questions simply get a normal answer.
Wrong matchesCalibration · 329 offered pairs
- Numbers only39/329
- With check0/329
True repeats offeredCalibration · 290 true repeats
- Numbers only290/290
- With check270/290
Unrelated questions offered a recipeSeparate full run · 508 questions
- Numbers only21/508
- With check0/508
Calibration only: the 20 dropped true repeats receive a normal answer. The separate full run below tests the selected threshold end to end.
| Measure | Numbers only | With meaning check |
|---|---|---|
| Wrong matches among offered recipes | 39/329 | 0/329 (threshold 0.1) |
| True repeats still offered a recipe | 290/290 | 270/290 |
| Unrelated questions given a recipe (full run) | 21/508 | 0/508 |
Separation: area under the curve 0.998; every wrong match scored at most 0.051, the median true repeat 0.953. The true repeats it drops are all romanised Indic.
View methods and limits
GSM8K-Indic questions that appear more than once in different languages; the first appearance is answered directly and with thinking and becomes a recipe only if both give the same number. Later appearances are answered three ways on identical questions, thinking off unless stated, greedy, on the same server. Significance is an exact paired test on the questions where two setups differ. The meaning check uses a public multilingual reranker; its threshold was calibrated on the first run and then confirmed in a separate full run. Maths only; one suite.
What this does not establishThe check, confirmed end to end
The threshold was chosen on the same data it was tested on, so the full evaluation was run again with the check switched on. Wrong matches on unrelated questions fell from 21 of 508 to 0, and with them the harm. Recipes were offered on 271 returning questions instead of 308, close to the 270 the calibration predicted, and NIF still beat the plain model significantly (23 fixed, 6 broken).
What this does not establish
These are maths questions, where a final number can be compared. Factual and open-ended questions need a different kind of verification, and a recipe for a fact can go stale when the world changes. Two answers agreeing is a weak verification; a user's confirmation or a source check would be stronger. The meaning check was calibrated on 329 pairs from one suite. Reuse across users raises a privacy question that this design answers only for non-personal content. Nothing here is used for answers in the chat service yet.
Sources and related work
The questions are the GSM8K-Indic frozen suite. The relevance model is a public multilingual reranker also used in Aditya's retrieval. Results come from saved evaluation records: three full runs of the same evaluation on two model versions, the calibration scores for every offered pair, and the run with the meaning check enabled. The agent-side NIF experiments, which motivated this work, are on the NIF research page.
Sources and related evidence
Frozen evaluation suite of translated grade-school maths problems, native and romanised Indic.
Internal sources -- three NIF-for-chat runs, calibration scores and the meaning-check run, October 2026.