The ASR heard English perfectly and wrote it in the wrong alphabet

A speech model I fine-tuned for Nepali worked. Word error rate went from 0.840 to 0.379, script accuracy from 70% to 100%. The write-up is here, and what it broke on the way is here.

Then real users started sending voice notes in English, and every single one came back like this:

spoken:  "implant network towers in"
model:   इम्प्ल्यान्ट नेटवर्क तावर्ज इन

Read that Devanagari out loud. It says implant network towers in. Every phoneme is correct. The model heard the sentence perfectly and wrote it in the wrong alphabet.

This post is the whole arc of fixing that: the metric that found it, the corpus that didn't exist, the experiment design that made the result trustworthy, the tokenizer bug that silently cost a nine-hour training run, and the A/B at the end that overturned what I thought I'd proved.

Word error rate cannot see this bug

The first problem is that WER is blind here. When the model writes English in Devanagari, every word is wrong, so WER pins at 1.0. When the model completely mishears ordinary Nepali, every word is also wrong, and WER pins at 1.0.

Same number. Two entirely different failures. One is an acoustic problem needing more data; the other is an output-vocabulary problem needing a completely different fix. You cannot tell them apart from the metric that your eval harness prints.

So the first commit was a metric, not a fix:

DEVANAGARI_CHAR = re.compile(r"[ऀ-ॣ॰-ॿ]")
LATIN_CHAR = re.compile(r"[A-Za-z]")

def script_of(text: str) -> str:
    """Which script this is written in: "deva", "latin", or "" for neither."""
    deva = len(DEVANAGARI_CHAR.findall(text))
    latin = len(LATIN_CHAR.findall(text))
    if deva == latin:
        return ""
    return "deva" if deva > latin else "latin"

Two details in that regex matter more than they look.

The Devanagari Unicode block (U+0900–U+097F) contains the danda ।, the double danda ॥, and the digits ०–९. My first version matched the whole block, so २०२६ counted as three votes for Devanagari. But २०२६ and 2026 are the same year written two ways — they say nothing about which language the model chose. The character class above deliberately skips U+0964–U+096F.

And a tie returns "", not a guess. An empty hypothesis has no script, and scoring it as a success would hide exactly the failure mode this metric exists to catch.

Then the harness reports it alongside WER:

lang           n      WER      CER   script  empty
en           599    1.013    0.910    0.000      2

Script accuracy 0.000. Not one clip in 599 came back in Latin. That's a far more actionable number than "WER is 1.0", because it says the acoustic model is fine and the output space is broken.

Diagnosis: it isn't mishearing, it's mis-spelling

Once you look at the outputs as transliteration rather than as errors, the picture inverts:

The encoder survived fine-tuning intact. What the fine-tune destroyed was the Latin branch of the output vocabulary — the model no longer believed Latin was a thing it was allowed to emit for those acoustics.

That reframing changes the size of the problem enormously. "Teach the model English" is a huge undertaking. "Restore an output branch it already had" is small, and predicts that a few hours of data should be enough. That prediction turned out to be right, and it's the reason the eventual fix used 8 hours rather than 8,000.

The mechanism is worth stating plainly, because it generalises: this model is a CTC architecture that takes no language hint at inference. It maps acoustics to one shared multilingual vocabulary. Nepali-accented English sounds like Nepali phonetics, so a model drilled to emit Devanagari for those phonemes emits Devanagari — regardless of which language the speaker believes they are speaking. Native-accented English still came back in Latin, because it sounds different enough to route elsewhere.

That last detail is why the bug survived the original testing. I had tested English. I tested it with clean, native-accent clips. Every real user of the product is a non-native speaker. The test population was the one group the regression doesn't affect. A small sample wasn't the flaw — an unrepresentative sample was.

The corpus that doesn't exist

To fix routing you need training examples of exactly the failing case: English audio, Latin transcripts, spoken in the target accent.

There is no such corpus. I checked properly, and the dead ends are worth listing because each one costs an afternoon:

• A 15,700-hour accented-English corpus distributed as a single split tarball across ~20 archive records of 50GB each. partaa is not a usable archive on its own, so there is no way to take a subset without pulling ~1TB. Its derivative on a model hub is text and metadata only — no audio.
• A promising-looking dataset whose card described English speech. 630 of its 642 transcripts were Devanagari. The card was wrong about its own contents.
• Common Voice mirrors, which are script-based loaders. The current datasets library dropped script support, so they no longer load at all.
• A conversational accented-English corpus with per-speaker L1 metadata — genuinely excellent, but with no speakers of the target L1, and no train split by design.

The lesson generalises past speech: verify a dataset's contents, not its description. Two of those four were eliminated by looking at the actual rows.

The find, and an experiment design that can't fool you

The answer was hiding inside an evaluation benchmark. A 9.6-hour accented-English corpus, built to test ASR systems, ships a primary_language column giving each speaker's first language. Grouping by it:

L1                clips   hours
Nepali              599    1.16   <- the largest single group
Kannada             577    0.71
Urdu                514    0.64
Malayalam           503    0.68
...  (19 L1s total,           9.61 h)

599 clips of native speakers of the target language speaking English, with clean English ground truth. Exactly the failing distribution.

The tempting move is to train on it. The better move is not to.

I held out the entire target-L1 subset as the evaluation gate and trained only on the other eighteen L1s. That converts the whole exercise into a real question: does training on other accents from the same language family teach the model to keep English in Latin for an accent it has never heard?

A random split would have leaked the target accent into training and answered a much easier question — one whose result would look great and mean nothing in production. Because a speaker has exactly one L1, splitting on L1 guarantees speaker-disjointness for free.

The mixture ended up as roughly 91 hours of the target language against 8.26 hours of English. Left proportional, English would be 8% of batches — too thin to move anything. The sampler exposes a temperature:

p_i ∝ n_i^β

β = 1.0   proportional        →  English 8%
β = 0.5   sqrt(hours)         →  English 23%
β = 0.0   uniform             →  English 50%

β = 0.5 puts English at 23%, which is enough to retrain an output branch without drowning 91 hours of the primary language in 8 hours of read English.

The bug that cost nine hours

The run finished. Script accuracy went 0.000 → 0.998. And the English WER was a mediocre 0.557 — far worse than the script number implied.

Looking at actual predictions:

"That"     -> <unk>hat
"English"  -> <unk>nglish

The tokenizer has no uppercase letters. It carries lowercase, Devanagari, and almost no punctuation — ' and - and that's it. The English corpus shipped ordinary written English: capitals, commas, full stops. 85% of the training targets contained characters the model is physically incapable of emitting.

So it learned to want them, and answered <unk>.

The existing data-prep for the primary language had always filtered to the tokenizer's charset. The English path was new code and didn't. The fix is unglamorous:

def normalise(text: str, vocab: set[str]) -> str:
    text = text.lower()
    return "".join(c for c in text if c in vocab or c.isspace())

Plus a --tokens flag that reads the tokenizer's vocabulary file and hard-fails if any target contains a character outside it. One file read against a nine-hour GPU run.

The re-run scored WER 0.260 — much better than the 0.354 I predicted by simply deleting <unk> from the old output. Unemittable targets weren't just appending junk tokens at the end; they were degrading the acoustic learning itself.

And a second thing fixed itself. In the buggy run, real Nepali clips had started coming back romanised — I'd written that up as the language boundary over-correcting toward Latin, and flagged it as a risk to watch. It wasn't. Nothing changed between the two runs except the training-target charset, and the romanisation vanished. The unemittable targets were blurring the routing. I'd blamed the objective when the data was at fault.

Results, and the A/B that overturned my conclusion

The gate — 586 held-out clips of target-L1 speakers speaking English, never trained on:

Training on eighteen other accents fixed a nineteenth the model had never heard. No target-accent training data was needed — which is fortunate, because none exists.

Then the part I nearly skipped.

I had a number saying the primary language was unharmed: 0.427 on the buggy run versus 0.418 on the fixed one. Slightly better. Comfortable.

But both of those runs had English in the mixture. Comparing them says nothing about what adding English cost. The honest comparison is against the original single-language model — and the two had never been measured on the same clips. One was scored on 200 utterances, the other on 3,840.

So I scored both models on 1,000 byte-identical clips:

model       WER      CER   script
v1        0.392    0.195    1.000
v3        0.429    0.214    0.999

delta:   +0.037  (v3 WORSE)

Adding English cost the primary language 0.037 WER — about 9% relative. The claim I'd been about to publish, that English was free, was wrong. It was free only against a comparison that couldn't detect the cost.

That's the trade, stated properly:

Worth taking here, because the majority of real voice notes are the accented English that the old model fails completely. But it is a regression, and if primary-language accuracy were the priority, the old model is better at it. Two models, two different right answers, depending entirely on your traffic mix.

The cheap version of this lesson: an A/B on identical inputs is worth more than two carefully-measured numbers from different runs. Subsampling to 1,000 clips made it cost about twenty minutes.

Shipping it

A model that scores well is not a model in production. Three things bit.

Multi-architecture builds. The staging box is ARM (Graviton); production is x86_64. A comment in the build workflow claimed both were x86_64. Acting on it, I dispatched an amd64-only build — an image that could not have started on staging at all. Caught by running uname -m on the host before deploying rather than trusting the comment. The comment is now a table of both hosts, and the rule is: never narrow platforms without checking the target.

Weights aren't in git. A 348MB model file is over the 100MB file limit, so it lives as a release asset that CI downloads and bakes into the image. Which means "is the new model live?" has three answers of increasing strength:

1. The /health endpoint reports a model name — but that's a build-time string, so it only proves which image is running.
2. The container's image digest and architecture — proves no stale cache or wrong platform.
3. md5sum of the model file inside the running container, compared to the file the training job produced — byte-level proof.

Only the third one actually answers the question. It's also the cheapest.

Deploys don't restart sidecars. The orchestrator treats accessories separately from the app: merging and deploying leaves the old model running indefinitely. Worse, the deploy sets the config flag that turns the feature on before the sidecar exists, so there's a window where the client is constructed and fails to connect on every request. It degrades to the old fallback path, but it burns a connect timeout each time. Merge and boot belong in one maintenance window, in that order, deliberately.

What I'd tell the next person

• If your metric can't distinguish two failure modes, it isn't measuring the bug. Script accuracy took ten lines and turned an unactionable "WER 1.0" into "the encoder is fine, the output space is broken."
• Test on the population you actually serve. I tested English with native-accent clips. Every real user is a non-native speaker. The sample wasn't too small; it was drawn from the wrong distribution.
• Hold out along the axis you're claiming to generalise over. Splitting by L1 rather than randomly is what makes "it transfers to unseen accents" a real claim instead of a leak.
• Validate training targets against the tokenizer's vocabulary before you spend the GPU. A charset check is one file read. Skipping it cost nine hours and produced a result that looked like a modelling problem.
• Compare on identical inputs. Two numbers from two runs on two different test sets cannot be subtracted, no matter how carefully each was measured.
• Verify the artifact, not the pipeline. Green workflows, health endpoints and image tags all describe intent. A checksum describes reality.

The through-line: every one of these was a case where something reported success while the thing underneath was wrong. Fine-tuning is mostly not a modelling problem. It's a measurement problem wearing a modelling problem's clothes.

Sizing a model service on a small box? The Latency Budget Calculator will tell you whether your inference fits inside the response time you're promising.