Building Voice of India:A national benchmark for Indic ASR

India does not speak the way ASR benchmarks assume. We interrupt ourselves. We switch languages mid-thought. We say "doctor" in the middle of a Hindi sentence and spell it in Devanagari one day, in Latin script the next. We pronounce numbers three different ways depending on register. And almost none of this looks like the clean, scripted, single-speaker English audio that most speech recognition systems are measured against.

For a long time, the Indic ASR field has had a quiet credibility problem. Models post single-digit Word Error Rates on academic test sets, then stumble the moment they meet real callers in real districts. The gap between leaderboard accuracy and production accuracy is not an engineering bug. It is a measurement bug. The benchmark itself is wrong.

Voice of India is our attempt to fix the measurement. It is a national, research-led benchmark covering 15 Indic languages and 1,200+ districts, built in partnership with AI4Bharat and IIT Madras, and scored using a new lattice-based protocol called OI-WER (Orthographically Informed Word Error Rate) that gives credit for linguistically valid variations instead of penalizing them.

This blog is the long version of how we did it: the data collection pipeline, the cluster-based sampling strategy, the six-stage transcription audit, the lattice protocol, and what the leaderboard actually tells us about the state of speech recognition for Indian languages today.

WER (Word Error Rate) measures the fraction of words in the reference transcript that the ASR system got wrong, computed as (substitutions + insertions + deletions) / total reference words. CER (Character Error Rate) applies the same formula at the character level instead of the word level, which makes it more forgiving of spelling and morphological variation — a useful trait for Indic scripts.

Word Error Rate was designed for English. It assumes every spoken word has one accepted spelling, that languages don't mix mid-sentence, and that the gap between formal and colloquial usage is narrow. Indian languages don't satisfy any of those assumptions. The result is that WER systematically misreads correct transcriptions as errors. Six failure modes show up over and over.

1. Code-mixing and the script-mismatch trap

A native Hindi speaker says "वह doctor के पास गया". The reference annotator kept doctor in Latin script. The ASR system transliterated it to डॉक्टर. Both are correct. Standard WER counts a substitution.

2. Colloquial vs formal registers

Every Indian language has a formal written register and a colloquial spoken register, both equally understood by native speakers. WER treats any deviation from the reference as an error.

3. Numeric format variability

The same number can be written three valid ways in an Indic language: as words (पांच सौ), as Arabic numerals (500), or in Devanagari numerals (५००). WER treats all three as unrelated tokens.

4. Orthographic variation: the Nukta problem

Loanwords from Persian and Arabic are routinely written with or without a Nukta (ज़ vs ज) depending on regional convention and publisher style. Both are correct. WER doesn't know that.

The core metric powering Voice of India is Orthographically Informed Word Error Rate (OI-WER). The idea is simple: instead of comparing the ASR output to a single reference string, we compare it to a lattice of linguistically valid alternatives for that utterance.

Each audio segment in Voice of India is mapped to a comprehensive set of valid lexical and phonetic variations: alternate spellings, code-mix variants, formal vs colloquial registers, common loanword choices, Nukta-on/off forms, and equivalent numeric renderings. OI-WER then computes the optimal path through that lattice — the single reference variant that minimizes edit distance against the model's output. If the model's transcription matches any linguistically valid variant, that segment is not penalized.

The lattice is not generated by an LLM judge at evaluation time. It is constructed offline, ahead of evaluation, by native-speaking linguists during a separate pipeline that runs alongside the primary six-stage transcription. This matters: it means OI-WER is fully deterministic, fully reproducible, and free of the model-drift problems that plague LLM-as-judge metrics.

The reference implementation is open source: AI4Bharat/OIWER on GitHub.

Voice of India does not use scripted prompts. The core dataset is built from dual-channel, unscripted telephonic conversations recorded across India. Dual-channel preserves acoustic isolation between speakers, allowing precise attribution. Unscripted preserves natural disfluencies, overlaps, code-switches, and the prosody of real Indic speech.

Most public Indic ASR benchmarks rely on read speech: speakers reading prepared sentences in quiet rooms into a clean microphone. That setting eliminates almost everything that makes real production audio hard — overlapping turns, interruptions, background TV, vehicle noise, mid-sentence code-switches into English, hesitations, laughter, and the long-tail prosody of unrehearsed speech. Models that score well on read speech routinely collapse on conversational telephony, which is exactly the channel where most Indian users actually interact with voice systems (customer support, field operations, IVR, voice assistants on entry-level devices).

We deliberately recorded over a narrowband telephonic channel to match this production reality. Each speaker is captured on an isolated channel, which means we can run speaker-attributed transcription and measure cross-talk cleanly. Conversations are recorded across 28 states and 8 union territories, spanning urban metros, tier-2 and tier-3 cities, and rural districts, so that accent, dialect, and channel conditions reflect the actual geography of Indian speech rather than a studio-controlled subset.

Raw conversations are programmatically segmented into utterances based on detected silences and speaker-turn boundaries. Every segment then passes a three-tier filtering protocol before it is considered for the benchmark:

1. Acoustic integrity. Signal-to-Noise Ratio thresholds ensure the speech is discernable while remaining challenging. We retain a small share of noisy audio (P808 ≤ 2.8) — typically 3–7% per language — because production systems have to handle it. Pure silence, DTMF tones, ringback, and hold music are filtered out automatically.
2. Cluster eligibility. The segment must come from a speaker registered against one of our 139 regional clusters, with verified demographic metadata (age band, gender, district). This guarantees that every utterance in the benchmark can be traced back to a known geographic and demographic slice, enabling fairness analysis across age, gender, and region.
3. Linguistic relevance. Segments are screened for primary-language content; pure-English segments and unintelligible audio are dropped before transcription. Natural intra-sentential code-switching (Hindi–English, Tamil–English, etc.) is preserved, because it is a defining feature of conversational Indic speech and a known failure mode for monolingual models.

The result is a corpus that looks much closer to what a deployed model actually encounters: short, overlapping, noisy, demographically diverse utterances drawn from real conversations rather than read scripts.

The six-stage gold-standard transcription pipeline

For every selected audio segment we run a machine-assisted, six-pass transcription and audit pipeline. The goal is near-zero ground-truth error.

1. Hypothesis generation. A primary transcription is produced by an ensemble of internal high-recall ASR models tuned for each of the 15 languages.
2. First linguist pass. A native-speaking linguist audits the hypothesis against the audio for absolute correctness.
3. Independent relay audit (round 1). A different linguist reviews the audio against the previous transcript without seeing prior reviewer notes.
4. Independent relay audit (round 2). A third linguist repeats the process. Any disagreement triggers a consensus discussion.
5. Variation extraction. A separate pipeline generates the lattice of valid lexical/phonetic variations for that utterance.
6. Lattice validation. Native-speaker reviewers approve, prune, or extend the lattice before the segment enters the benchmark.

Each segment passes through at least three independent native-speaker reviewers before it is considered gold standard. This is expensive. It is the reason the benchmark exists.

A common failure mode in Indic benchmarks is to treat every region as equal. Voice of India does the opposite. We use Population-Proportional Stratified Sampling across 139 regional clusters defined to capture distinct phonetic and lexical signatures. The volume of data sampled from each cluster is determined by the actual population percentage of that region — so the dataset's demographic weight matches the country's.

That single choice changes everything. It means a Telugu speaker from coastal Andhra and a Telugu speaker from Telangana are both represented in proportion to their real populations, instead of being averaged out into a single "Telugu" bucket. It means Hindi covers 306 districts, not three studio cities. And it means the score a model gets is closer to the score it would earn in production.

Why geographic representation actually matters

A single headline number like "Tamil WER = 20%" is one of the most misleading artifacts in Indic ASR. It collapses a country-sized distribution into a single point estimate and hides the only signal that matters for production: where the model is failing. The same model reported at 20% WER on Tamil might in reality be operating at 10% WER in Chennai and central Tamil Nadu — fluently handling the dialect it has seen most during training — while degrading to 60–80% WER in Tirunelveli, Kanyakumari, or among Sri Lankan Tamil speakers, where phonetic cadence, vowel length, and loanword inventory shift sharply. The 20% average is real, but no user actually experiences 20%. They experience either the 10% version of the model or the 80% one, depending entirely on where they live.

Population-proportional clustering is what makes this visible. By breaking each language into its real geographic and demographic strata — and weighting the dataset so each cluster contributes in proportion to its actual speaker population — we can decompose a single WER number into a per-cluster distribution. Companies deploying voice products can then see exactly which districts, age bands, or gender cohorts are being underserved, prioritize targeted data collection or fine-tuning for those clusters, and track whether quality improvements are uniform across the country or concentrated in already-strong regions. The goal is not just to lower the headline number, but to flatten the variance — so a user in Bastar gets the same product quality as a user in Bengaluru.

Per-language dataset breakdown

536 hours of curated audio across 306,230 utterances and 36,691 unique speakers. Hindi spans 306 districts; Bhojpuri and Urdu pull from 99 and 119 respectively.

The shape of the dataset matters as much as its size. Telugu and Bengali concentrate volume into deep speaker pools; Hindi spreads thin across 306 districts to surface regional variation; Urdu and Bhojpuri intentionally over-index on district coverage relative to their hours, because dialectal drift in those languages is the entire point of measurement.

The collection pipeline, end to end

Eliciting spontaneous Indic speech at scale is harder than collecting scripted prompts. To draw out natural, extended speech, we curated a repository of 1,000+ conversational topics per language — everyday life, finance, healthcare, agriculture, education, travel — generated initially by Gemini 3 Pro and then audited by native language experts for cultural fit. Each topic surfaces as an open-ended cue followed by progressively revealed follow-up questions that pull speakers toward richer descriptions.

Audio is captured over internal infrastructure that preserves dual-channel audio (speaker isolation), with WebRTC VAD performing initial segmentation. Adjacent speech regions are merged across short silences and length-filtered. Automated language identification (Meta MMS) flags off-language segments before they reach human review. Every contributor is screened for language familiarity before being granted recording access; all participants provide informed consent under an institute-approved ethics protocol.

For Voice of India to remain a useful long-term measurement, it has to stay uncontaminated. Once a benchmark leaks into training pipelines, its scores stop measuring generalization and start measuring memorization. The history of NLP benchmarks is largely the history of this exact failure.

So the core evaluation set is held private. No model provider sees the held-out audio or the lattice references.

We will release a public sample set for each of the 15 languages, covering the full pipeline — the audio, the gold-standard transcript, the lattice variants, and per-model outputs. The public set is large enough to debug a model and understand the lattice protocol, and small enough that it cannot meaningfully contaminate training.

We evaluated 16 models against Voice of India: open-source releases from Meta, AI4Bharat, and Google, alongside proprietary APIs from OpenAI, Google, Microsoft, AWS, Deepgram, AssemblyAI, ElevenLabs, and Sarvam. The aggregate ranking is a corpus-level OI-WER computed across languages, pooled by reference word count — so models can't game the score by performing well only on smaller subsets.

Drilling per-language shows where the leaderboard's averages mislead. Hindi is the most competitive language, with multiple systems under 8% OI-WER and the best near the noise floor of human inter-annotator agreement. Tamil, Malayalam, Telugu, and Kannada remain hard for almost everyone — agglutinative morphology and dialectal density push even strong proprietary systems into the high teens. Sarvam's models lead on most languages; Gemini 3 Pro is the strongest non-Indic- specialised system; open-source IndicConformer is competitive with proprietary APIs on Hindi, Bengali, and Urdu.

Model coverage is not uniform. Several proprietary systems do not support Assamese, Maithili, Odia, Punjabi, or Gujarati natively. We do not extrapolate scores for unsupported languages — a missing cell on the leaderboard is a missing capability, not a low score.

Public benchmarks like FLEURS are widely used to report Indic ASR performance, but their static, publicly accessible evaluation sets are vulnerable to overfitting. Models tuned to the leaderboard exploit benchmark artifacts; their scores stop measuring generalization. Voice of India, scored on held-out telephonic audio, exposes the gap.

The reordering matters as much as the absolute jump. On FLEURS, Deepgram Nova 3 (16.7%) and GPT-4o Transcribe (9.1%) appear competitive. On VoI, both fall to the bottom of the leaderboard while Sarvam Audio — undervalued by FLEURS at 8.2% — emerges as the actual leader at 13.0% lattice WER. Single-reference WER on the same audio inflates errors further (15.4% for Sarvam, 40.3% for GPT-4o), confirming that a meaningful slice of the “errors” older metrics flag are valid orthographic variations, not transcription failures.

Aggregate WER hides the conditions under which models actually fail. We sliced the evaluation set across DNSMOS quality quartiles and speaking-rate buckets to see what production teams should actually expect.

Across every system, error rate falls monotonically as recording quality improves — with roughly a 2× gap between the worst quartile (Q1) and the best (Q4). For production deployments where audio quality is uncontrolled, the worst-quality decile of calls will dominate aggregate WER.

Speaking rate produces a clean U-curve. Both very slow speech — typically hesitant, disfluent, or pause-heavy — and very fast speech increase error rates noticeably. The optimum sits inside the “normal” band at roughly 4-5 words per second. Anything outside that window costs accuracy, regardless of vendor.

Audio duration shows a similar shape: very short (<2s) clips lack acoustic context for disambiguation, and very long (>5s) clips amplify cumulative drift. The 2-5 second sweet spot mirrors what production VAD pipelines tend to produce anyway.

Reporting one number per language averages out the most important signal: where in India a system fails. We computed average WER per district across four pan-India models supporting all fifteen languages.

District-level WER ranges from ~4% (Nainital, in the Himalayan Hindi belt) to ~44% (Mannarakkat in interior Kerala). The Hindi belt — Uttar Pradesh, Delhi, Haryana, Rajasthan, Madhya Pradesh — clusters under 10% WER. Metropolitan districts also tend to perform well. Linguistically diverse or underrepresented regions, especially North Bihar (Maithili, Bhojpuri) and interior Kerala (Malayalam), exhibit substantially higher error rates. The pattern is a clear geographic bias and a direct consequence of training-data distribution, not modeling capacity.

For out-of-region migrants, the picture is worse. A Chattisgarhi speaker calling from Tamil Nadu — a real, common production scenario — sees WERs of 55-65% across most systems. ASR robustness is not a property of the language; it is a property of the speaker-and-region pair.

One of the most striking findings in the per-attribute analysis is a consistent gender gap. Across architectures, vendors, and languages, male speakers see roughly 20% relative WER inflation versus female speakers on the same content.

The effect is too consistent to be coincidence and too large to ignore. The most likely cause is upstream: female-skewed Indic training corpora over the last several years have produced acoustic models that fit female pitch ranges and articulation patterns more tightly. Closing this gap requires deliberate balancing in pretraining audio, not architectural changes.

Five patterns are worth flagging:

1. The Hindi gap is closing fast — the South is not. Best-in-class OI-WER on Hindi is now under 5%, near the noise floor of human inter-annotator agreement for spontaneous telephonic speech. The same trajectory is not visible in Tamil, Malayalam, Telugu, or Kannada, where OI-WER stays 2–4× higher across almost every system and every architecture. The South-Indian ceiling is structural, not a tuning problem.
2. Indian ASR is not "solved." The popular narrative — driven by single-digit WER on a handful of public leaderboards — overstates how usable these models actually are in production. The same systems that look saturated on public benchmarks degrade sharply on unseen speakers, unseen districts, and natural code-switched speech. Companies betting deployment decisions on those leaderboards are buying a number, not a capability.
3. Public benchmarks do not reflect real-world deployment. Most open evaluation sets have been around long enough to have leaked into pretraining corpora, been overfit on, or been tuned against. They cover a narrow slice of speakers, channels, and domains, and they do not surface the edge cases — telephony noise, accent drift, named entities, numerals, code-switches — that dominate failures in the field. A model that wins a public benchmark and a model that survives a call-center floor are not the same model.
4. After a point, geography is the lever — not architecture. Once a model is broadly pretrained on a language, further gains stop coming from bigger encoders or fancier objectives. They come from going to specific districts, recording the dialect that actually lives there, and folding that audio back into training. District-level WER (Chhapra at 44%, Gorakhpur at 38.5%, vs. Nainital at 4%) is the signal that tells deployment teams where the next percentage point of real-world quality has to come from.
5. Single-reference scoring unfairly penalizes Indian languages.Standard WER assumes one canonical transcript. Indian languages routinely admit multiple equally valid spellings, sandhi joins, transliterations, and morphological variants for the same utterance. Scoring against one reference treats legitimate variation as error. A closed set of accepted lattice variations per utterance — the approach OI-WER takes — is what makes a benchmark honest about Indian speech instead of artificially harsh on it.

Voice of India is a living benchmark. The roadmap focuses on the failure modes that still hide inside aggregate OI-WER:

• Named-entity coverage. Dedicated evaluation of place names, person names, and organization names — the parts of a transcript that carry the most downstream weight in production systems.
• Domain-specific evaluation sets. Healthcare, finance, education, governance, and public-services audio, scored separately so deployment teams can see ASR readiness in their actual domain.
• Numerical and semiotic segmentation. Finer-grained scoring for dates, currencies, measurements, and percentages — the categories where ASR failures most often translate into business failures.
• Expanded language coverage. More languages, more dialect clusters, and longer-form audio.

Voice of India exists because of the native-speaking transcribers, reviewers, and linguists across all 15 languages who did the slow, careful work that no automated pipeline can replace. It exists because of AI4Bharat and IIT Madras, who lent decades of Indic NLP research to the design of OI-WER. And it exists because we think the conversation about Indian-language AI deserves a measurement that India can recognize as fair.

To request an evaluation against the private set, write to voi-evals@joshtalks.com.