Skip to content
← Journal

The two words our OCR model got wrong were right

Our reading model disagreed with the volunteers' proofread text twice on a held-out page. Both times, the printed page agreed with the model.

One page we never trained on

When we train our document-reading model on public-domain books from Wikisource, we hold a few books out completely. The model never sees a page from them, so they tell us how it reads print it has not met. One of them is the third volume of Saraswatichandra, the Gujarati novel by Govardhanram Tripathi, printed in the 1890s.

On page 145, a block of six printed lines, the latest model (v6) wrote 59 words. The volunteers who proofread the book wrote 59 words too. They differed in two places.

Two disagreements

Where the proofread text says દંભ (pretence), the model wrote દમ્ભ. Where the proofread text says વગેરેનો (of the rest), the model wrote વગરેનો. By the usual measure, those are errors, and they account for all of the model's character errors on this page: three edits in 351 characters.

The obvious conclusion is that the model misread two words. We almost wrote it down that way.

The same printed word, three ways
An unsupported leap

Proofread text: દંભ · Older model: દરભ

A bounded conclusion

Latest model: દમ્ભ, as printed on the page

Then we looked at the scan

The printed page says દમ્ભ, with the nasal written as a full letter and a half-letter joined to the next one: the older spelling. It says વગરેનો without the vowel sign the modern spelling adds. The model wrote exactly what was printed. The volunteers, quite reasonably, wrote the words the way a modern reader expects them.

Neither side is wrong. A proofreader making a book readable today and a model asked to transcribe a page are doing different jobs, and on this page the two jobs disagree by two words.

What the older model did

An earlier version of the model, v4, read the same page with twice as many character errors. It also wrote વગરેનો, following the print. But it made three real mistakes: a wrong first letter in યુદ્ધપ્રસંગે, a garbled દરભ, and a city name, રત્નનગરી, with letters dropped.

So the improvement between the two versions on this page is not the two words in question. It is the three real misreadings that went away.

What this means for the score

Every benchmark compares a model's output with a reference text written by people, and people normalise. They modernise spelling, expand abbreviations, fix obvious misprints, and sometimes write numbers as digits and sometimes as words. A model that copies the page faithfully is penalised whenever the reference did not.

That does not make the benchmark useless. It means a small difference in score can be a disagreement about conventions, not about reading. Before believing that a model got worse or better by a fraction of a point, it is worth reading the errors themselves.

Speech has the same problem

We found the same thing in speech recognition. A Hindi speaker reads a sentence that mentions 1200 GMT. The model writes the time and the acronym the way they were spoken, in Devanagari words. The reference writes digits and Latin letters. A transcription with no mistakes is scored at almost one word in five wrong.

In both cases the fix is the same habit: report the error rate, then read a sample of the errors, and say which ones are conventions.

What we changed

We did not change the model to match the volunteers. For a reading tool, faithfulness to the page is the right default: a historian or a lawyer wants what was printed, not a modernised version. Where a modern spelling is wanted, that is a separate, optional step after reading.

We did change how we report results. Our research pages now show real model output next to the reference, mark which differences follow the page, and keep the full per-language numbers available for anyone who wants to check.

One page is an example, not a result

This is a single page, chosen because it shows the problem clearly. It does not tell you how often the model's disagreements with references are convention rather than error; measuring that would mean checking a large sample by hand against the scans. Our benchmark numbers are reported as measured, with the references as they are.

Further reading

Reading 22 Indian scripts: the full OCR reportSpeech to text, and what the score hidesModel Context Protocol — Tools specification