Benchmarks
Sanskrit audio-to-text recognition: own-benchmark leaderboard (in-distribution) and public benchmark (Shrutilipi, real speech) in progress.
Own-Benchmark: Held-Out Specialist Corpus
Task: Convert held-out Sanskrit audio → romanized text (lower CER is better).
Audio source: Native Sanskrit (indic-parler-tts TTS voice) — 58 clips, controlled phonetic coverage.
Metric: CER (character error rate) under the "ASCII phonetic skeleton" scheme — transliteration-folded, scheme-neutral, fair to all models.
⚠ Caveat: Own-Benchmark Scope
This leaderboard measures in-distribution performance: the same TTS voice for all models, identical audio, identical scoring. It shows the specialist ALM's strength on a controlled task. It does NOT claim to measure general Sanskrit ASR or real-world robustness. The public benchmark (Shrutilipi, real human speech) will test generalization.
✓ How We Fixed the Bugs
Earlier runs fed Qwen2-Audio no audio (the audio= kwarg was silently ignored in transformers 4.57 — a harness bug). That produced a single canned hallucination (CER 15.9 vs 0.44 now). We fixed it to pass audio explicitly. Our specialist also had a gold-length oracle; we retrained with free decode + EOS stop. These corrections are the honest exemplar: errors found, fixed, reported.
| # | Model | Params | Type | CER (norm) | Note |
|---|
Full per-clip predictions: data/benchmark/alm_vs_alm_records.json. The correction itself (fixing harness bugs) is the honesty exemplar for benchmarking.
Public Benchmark: Shrutilipi-Sanskrit (Real Human Speech)
Evaluation against real Sanskrit speech corpus (Shrutilipi, by community contribution). This tests robustness beyond TTS, real accent variation, and generalization.
Public Benchmark: LibriSpeech (English) — where we lose
The hypothesis was that one sphoṭa-principled model could stand on English too. It cannot: on the standard English ASR benchmark Śabda-ALM is last of four, by a wide margin. We show it in full rather than omit it. Why is the interesting part — see the interference finding in the paper (§3.2): the frozen core models English fine on its own, but a single shared adapter cannot serve both languages, and Sanskrit captures it.
Combined Bilingual Leaderboard
Macro-averaged word error over the public test sets every system ran, so the comparison is like-for-like. Ranking by all sets a model happened to run would put us first purely because we are the only system that also ran the chant set — a coverage artifact, not a result; we rank on the common set instead. We place second, not first — Whisper-large-v3 tops the combined board, because its near-perfect English outweighs its moderately-worse Sanskrit. Our win is specifically the real-Sanskrit board above.