Vedic v1: speech to text in Indian languages, and what the score hides
How we turned an open speech model into a no-settings product for 22 Indian languages and English, and why word error rate misleads in these languages.
Updated
Measured on locked test sets (November 2026). The acoustic model is AI4Bharat's IndicConformer; our work is the routing, the language choice and the measurement.
Speech to text · what the score hides
Model heardवाहन को उसी दिन तकरीबन बारह सौ जीएमटी के समय पर दुर्घटनास्थल से दूर ले जाया गया
Referenceवाहन को उसी दिन तकरीबन 1200 GMT के समय पर दुर्घटनास्थल से दूर ले जाया गया
18.8% word errors
20.7% character errors
for a transcription with no mistakes. “1200” and “बारह सौ” are the same number; “GMT” and “जीएमटी” the same letters.
FLEURS (Google), Hindi test set, clip 381; CC BY 4.0
In this paper 11 sections
Speak, and get the right words back
People in India switch languages without announcing it. A speech-to-text tool that first asks which language you are speaking has already failed most of them. We wanted something simpler: press record, speak Hindi, Tamil, Gujarati or English, and get the words back in the right script, with nothing to choose.
We did not train a speech model from scratch. AI4Bharat has already released IndicConformer, a 600-million-parameter model that transcribes all twenty-two scheduled languages under an MIT licence. Vedic v1 is what we built around it: a way to decide the language from the audio alone, a separate path for English, and a careful look at what the usual score does and does not tell you.
Three decisions before a single word
First, is this English? An open language-identification model (VoxLingua107, from SpeechBrain) gives a probability for each of 107 languages. Our users speak an Indian language or English, so the question is narrow: does English beat every Indian language? If it does, the audio goes to NVIDIA's Parakeet, an English model that also restores punctuation.
Second, which Indian language? IndicConformer's output layer has a separate block of 256 tokens for each language. Run once without choosing, the share of probability that lands in each block is a language guess the model makes for free. Hindi and Urdu are pooled, because spoken they are one language and differ only in script; we write Devanagari by default.
Third, transcribe in that language with the model's stronger decoder. The whole path runs on an ordinary CPU, at a small fraction of real time.
Three test sets that sound different
IndicVoices (AI4Bharat) is read and conversational speech recorded across many districts. FLEURS (Google) is people reading Wikipedia sentences aloud. Vaani (ARTPARK and IISc) is spontaneous speech from district villages, often over a phone. We froze a fixed sample of each, locked by checksum, and never train on any of them.
They are deliberately different. A model that does well on read sentences can still struggle with a farmer describing a crop over a bad line, and the second is closer to what a real product hears.
Two error rates, three very different pictures
Given the right language, the model's word error rate is 17.1% on IndicVoices, 18.4% on FLEURS and 43.2% on Vaani. Its character error rate on the same audio is 5.6%, 7.1% and 13.7%.
The two measures disagree by a factor of three, and the reason matters. Word error rate counts a whole word wrong for one wrong vowel sign, a different word break or a different but valid spelling. In languages with long, joined-up words, that happens constantly. Character error rate is closer to what a listener would call a mistake.
%
IndicVoices1,800 clips
- Word error rate17.1%
- Character error rate5.6%
FLEURS1,800 clips
- Word error rate18.4%
- Character error rate7.1%
Vaani1,200 clips
- Word error rate43.2%
- Character error rate13.7%
Word error rate counts a whole word wrong for one wrong vowel sign or a different word break; character error rate counts the characters. Vaani is spontaneous district speech and is much harder on both.
View methods and limits
Locked samples of IndicVoices, FLEURS and Vaani test splits (checksums fixed before any run). Word and character error rate after the scorer’s normalisation (punctuation and case removed), pooled over all clips of a set. “Language given” tells the model the right language; “no language given” runs Vedic v1 exactly as the product does. Hindi and Urdu are pooled for language identification. CPU timings on one desktop processor.
What this does not establishA perfect transcription scored as wrong
The clip at the top of this page is the clearest example we found. A Hindi speaker reads a sentence about a vehicle moved at 1200 GMT. The model hears every word and writes the time and the acronym the way they were spoken, in Devanagari words. The reference writes digits and Latin letters. The standard scorer counts three word errors and a higher character error rate, for a transcription a Hindi reader would call perfect.
It is not one clip. On FLEURS, ignoring the words that are numbers moves the word error rate from 18.4% to 15.2%. Some of what looks like mishearing is a disagreement about how to write a number.
Language by language
By word error rate, Tamil and Telugu look much worse than Hindi or Marathi. By character error rate the gap narrows a great deal. Both languages build long words from many parts, so one slip in a long word costs as much as missing a short one entirely.
This is why we report both numbers for every language, and why a claim that a speech model is twice as good in one language as another deserves a second look at how it was measured.
By words, Tamil looks more than twice as bad as Hindi; by characters the gap is much narrower. In languages with long, joined-up words, one small slip costs a whole word.
View methods and limits
Locked samples of IndicVoices, FLEURS and Vaani test splits (checksums fixed before any run). Word and character error rate after the scorer’s normalisation (punctuation and case removed), pooled over all clips of a set. “Language given” tells the model the right language; “no language given” runs Vedic v1 exactly as the product does. Hindi and Urdu are pooled for language identification. CPU timings on one desktop processor.
What this does not establishA fine-tune that learned a spelling, not an ear
Vaani transcribes colloquial forms the way people say them, where other test sets use the written standard. We fine-tuned a smaller Telugu model on Vaani to see whether it would hear district speech better. On Vaani, word errors fell from 41.3% to 34.5%, which looks like a clear win.
It was not. On IndicVoices the same model got worse, from 23.1% to 31.7% word errors, and character errors moved only a little on either set. The model had learned Vaani's conventions for splitting and spelling words. We now judge every speech fine-tune by character errors on more than one test set, never by word errors on the set it trained on.
Vaani’s word errors fell sharply, IndicVoices got worse, and character errors moved little, in opposite directions: the model learned Vaani’s way of splitting and spelling words, not better hearing.
View methods and limits
Locked samples of IndicVoices, FLEURS and Vaani test splits (checksums fixed before any run). Word and character error rate after the scorer’s normalisation (punctuation and case removed), pooled over all clips of a set. “Language given” tells the model the right language; “no language given” runs Vedic v1 exactly as the product does. Hindi and Urdu are pooled for language identification. CPU timings on one desktop processor.
What this does not establishNo language picker: what it costs
Vedic v1 has no language setting, so the honest test is to give it no language. Across all 4,800 test clips it chose the right language 4,588 times. Word error rate without a language was 19.0% on IndicVoices, 18.8% on FLEURS and 44.3% on Vaani, against 17.1%, 18.4% and 43.2% when the language is supplied.
The first version of the rule cost more. Almost every wrong choice was the same one, another language heard as Hindi, because Hindi and Urdu were pooled by adding their shares, which tipped close calls. We recalibrated on speech from a separate corpus that appears in no test set: the pooled Hindi share is now divided by 1.5 before the vote. On the test sets, read once after that choice, wrong languages fell from 351 to 212. The cost: a few more Hindi clips are now heard as something else.
Every number elsewhere on this page tells the model the language first. A product never gets that help, which is why this comparison matters more than any single score above. Where the language is already known, passing it removes the remaining cost.
Hindi and Urdu count as one spoken language (written in Devanagari by default), so an Urdu guess on Hindi speech is counted as right.
View methods and limits
Locked samples of IndicVoices, FLEURS and Vaani test splits (checksums fixed before any run). Word and character error rate after the scorer’s normalisation (punctuation and case removed), pooled over all clips of a set. “Language given” tells the model the right language; “no language given” runs Vedic v1 exactly as the product does. Hindi and Urdu are pooled for language identification. CPU timings on one desktop processor.
What this does not establishEnglish, in English
Early versions sent English speech through the Indian-language model, which wrote it phonetically in Devanagari. The English gate fixes that by asking only whether English beats every Indian language, not whether English beats all 107 languages it knows: accented English often scores as French or Welsh, but almost never as Tamil.
On our checks, none of 180 Indian-language clips was sent down the English path, and synthetic English clips came back word for word. Indian-accented English from real speakers is the case we have tested least, and the one we most want people to try.
What this does not establish
These are sampled, locked test sets for six to ten languages each, not all twenty-two. The languages without a test set here are transcribed by the same model but are unmeasured by us. Background noise, overlapping speakers and long recordings are not covered.
The acoustic model is AI4Bharat's, the English model NVIDIA's, and the language-identification model SpeechBrain's. Our contribution is how they are combined and how the result is measured. A better score on these sets does not by itself mean a better experience for every speaker.
Sources and evidence
All three models are public, as are the three test corpora. Our per-set, per-language error rates are in the downloadable evidence file, generated directly from the saved test runs.
Sources and related evidence
The acoustic model for all 22 languages; MIT licence
English transcription with punctuation; CC BY 4.0
Decides English versus the Indian languages; Apache 2.0
FLEURS (CC BY 4.0), IndicVoices (AI4Bharat) and Vaani (ARTPARK-IISc); test use only
Word and character error rates per test set and language