Catastrophic forgetting: what fine-tuning really costs

The result was good. Word error rate on the target language went from 0.808 to 0.379, and the model stopped writing that language in the wrong alphabet — 75% correct script to 100%. The full write-up is here.

This post is about the three things that went wrong around it. They're more useful than the win, because each one produced a green signal while something was quietly broken, and in all three cases I had measured something — just not the thing that mattered.

Failure one: it learned one language by forgetting others

After deploying, someone happened to try a German clip.

[german]   base:       alles hat ein ende nur die wurst hat zwei
           fine-tuned: अलस हात एन एन्ड नुदि वोस्थाथ सवाै

[spanish]  base:       no preguntes que puede hacer tu país por ti
           fine-tuned: उन प्रेगुन्तस के पेदै आसेरतो पाइस

German and Spanish were coming back transliterated into Devanagari — the alphabet I had just spent a training run teaching the model to prefer. English and French survived, degraded but readable.

This is catastrophic forgetting, and it's well documented. It happens when fine-tuning updates weights that other capabilities depend on. My training config had freeze_encoder_for_n_steps: 0, meaning the shared audio encoder was updated from the very first step rather than only the language-specific output layer. I bought one language by spending general multilingual competence.

What makes it a genuine incident rather than a known trade-off is that it was invisible to every metric I had. My evaluation suite measured the target language, thoroughly and correctly, on 3,840 held-out utterances. It said the model got dramatically better. It was right. It simply had nothing to say about the languages I wasn't testing, and absence of a metric reads exactly like absence of a problem.

I found this by chance. That's the actual finding. A regression discovered because a colleague idly tried a German sentence is a regression that could just as easily have reached users.

The fix isn't better luck:

[english]  devanagari=0%
[french]   devanagari=0%
[german]   devanagari=86%   *** REGRESSED ***
[spanish]  devanagari=100%  *** REGRESSED ***

2 regression(s)                                    exit=1

A script that runs identical clips through both models, computes what fraction of each output is in the new alphabet, and exits non-zero if a non-target-language clip crosses a threshold. Twenty lines. It now gates deployment.

Whatever you traded away, write the test that detects it. You already know what you risked — you chose it. The test is cheap; noticing later is not.

Failure two: the principled fix, measured and rejected

The textbook answer to catastrophic forgetting is to freeze the shared encoder and train only the output head. The base capabilities can't drift if the weights holding them don't move.

So I ran it again. Same corpus, same 5,000 steps, same held-out set, one config line changed. Another free GPU session.

Freezing recovered roughly 40% of the gain. And its validation curve went flat rather than running out of steps — it wasn't undertrained, it was capped.

That result is informative in itself. It says the original failure was acoustic-to-script, not vocabulary bias. The encoder had to move, because the mapping from sound to the right alphabet lived in the encoder. A head-only fix can't reach it, which is exactly what "40% of the gain and then a plateau" looks like.

It also didn't fully deliver what it was supposed to buy. 97% correct script with 103 empty transcripts means the shared output head drifted toward the new alphabet regardless — the head is shared across every language too, so training it on one language biases it no matter what the encoder does.

I kept the unfrozen model. The languages freezing protects aren't the ones this product serves, and paying 0.28 absolute error plus a 1-in-33 chance of an empty transcript to protect them is a bad trade here.

The part I want to underline: I wrote up the negative result instead of deleting it. It sits in the engineering log with its numbers. The next person to ask "why didn't you just freeze the encoder?" gets an answer with data rather than a shrug. A measured failure is a permanent asset; an unmeasured hunch is a conversation you keep having.

Failure three: the load test that tested nothing

The service needed to fit on a small box already running six other containers. So I load-tested it.

I generated audio with ffmpeg — sine tones at various lengths — fired twelve requests, watched memory stay under the cap, and called it verified.

It verified nothing.

Tones decode to near-silence. The recognizer's decode path exits early on them, so the compute-heavy branch barely runs and the memory arena never grows the way real speech grows it. I had measured the service's behaviour on input that doesn't resemble its input.

And I'd watched the wrong thing. I sampled the container. The question that decides your instance size is whether the other services still have room while this one works — which is a property of the machine, not the container.

The real test used genuine speech from held-out speakers, concatenated into message-shaped clips of 6 to 181 seconds, encoded exactly as a real voice note would be. 45 requests, 1,200 seconds of audio, up to 4 concurrent, sampling the whole machine every three seconds plus a latency probe against the neighbouring application.

memory        cold 497 MB -> steady 711 MB, peak 720 MB
33 concurrent requests moved steady state by 2 MB
available     never below 311 MB;  under 5 MB paged
co-tenant app p50 5 ms, max 43 ms, 0 errors in 148 probes

The conclusion flipped the plan. Memory was never the constraint — it's flat, and bounded by a config knob. The real limits are CPU credits on a burstable instance and queue latency, and the bigger instance I'd been considering fixes neither, because it accrues credits at the same rate.

Two things I nearly got wrong inside that test:

Swap usage is not swap pressure. The box sat at 415 MB of swap throughout and I almost upgraded it over that number. The signal is the rate — pages moving in and out per second — which was flat. That's memory parked earlier, not memory being fought over now.

The first measurement after a deploy is noise. My first end-to-end timing showed 18 seconds for a 6-second clip. Steady state was 3.65 seconds. Had I reported the first figure, everyone downstream would have had a latency budget three times too pessimistic.

The pattern in all three

Each failure came from measuring what was easy to measure and treating the green result as coverage.

• I measured the target language exhaustively. I didn't measure what I traded away.
• I measured memory with convenient synthetic input. I didn't check the input resembled reality.
• I measured the container. I didn't measure the machine the container had to share.

None of these were skipped tests. Each produced a number that looked fine. That's the dangerous shape: a silent gap in coverage is indistinguishable from a pass, and the more diligent your measured area, the more confident you are about the unmeasured one.

The habit I've taken from it is a single question before trusting any green result:

What would this test still say if the thing I'm worried about were broken?

For the language suite: unchanged — it doesn't touch other languages. For the tone load test: unchanged — tones don't exercise the decode path. For the container-only sampling: unchanged — it can't see the neighbours.

Three greens, three blind spots, one question that would have caught all of them. It's the same question worth asking of any production readiness checklist before you rely on it.

The honest scoreboard

What shipped is a specialist, not an upgrade. It's dramatically better at one language and measurably worse at several others. That's the correct trade for this product and would be the wrong trade for a general transcription service, and the difference between those two sentences is the entire value of having measured it.

Still open, and written down rather than quietly forgotten:

• The English evidence is four clips, not a corpus. English is the language users actually code-switch into, so a real English set gates any wider rollout.
• The frozen model was never run through the multilingual check — the decision stopped depending on it, so the check was skipped. That's a known gap, labelled as one.
• The next lever is mixing a fraction of multilingual data into training, which attacks the cause instead of disabling the part of the network that was doing the work.

The model result was the easy part. Knowing precisely what it cost took three more experiments, two of which I'd rather not have had to run.