MLOps for fine-tuning a speech model on a free GPU

I fine-tuned a multilingual speech recognition model — Meta's Omnilingual ASR, the 300M CTC variant — to handle a low-resource language. The training run that worked took 5.4 hours on a free-tier T4 and cost nothing. It was attempt nineteen.

The eighteen before it failed on environment resolution, asset registration, checkpoint paths, an upstream crash, and one memorable case where the install reported success and the module didn't exist. Not one of them failed on machine learning. They failed on the boring apparatus around it, which is where the real work in this kind of project lives.

This post is about that apparatus. If you want the model result — word error rate 0.808 to 0.379 — it's here. If you want what it cost, that's here. This one is about why run nineteen was trustworthy.

Build the evaluation harness before the training script

The single highest-leverage decision was writing the eval first and making it import the same class the server uses.

from app.asr import Recognizer   # the exact class the API serves from

rec = Recognizer()
for row in manifest:
    hyp = rec.transcribe(read(row["audio"])).text
    pairs.append((row["text"], hyp))

A benchmark that measures different code than you ship is worse than no benchmark, because it produces a number you'll trust. Reusing the production recognizer means quantization, audio decoding, resampling, and chunking are all inside the measurement. When the number moves, it moves for the reason you think.

This is not a hypothetical risk. A 2026 benchmarking effort for this language found earlier published results had been corrupted by a float16/float32 mismatch that silently produced empty predictions — and therefore a perfect-looking 100% error rate that meant nothing. Any harness that only reads a metric, and never a transcript, will happily report that forever.

Make the split reproducible without storing it

Speech data has a trap: the same speaker appears in many clips. Split randomly and a voice lands in both train and test, so the model gets credit for recognizing a person rather than a language. Your numbers look great and don't survive contact with a new user.

The split has to be speaker-disjoint. The lazy way is to shuffle speakers with a seed and save the assignment file — which then becomes a build artifact you must version, ship, and never lose.

Instead, hash:

def stable_bucket(key: str) -> float:
    """Deterministic [0, 1) — reproducible splits without storing a seed."""
    digest = hashlib.sha256(key.encode("utf-8")).digest()
    return int.from_bytes(digest[:8], "big") / 2**64

# speaker -> split. Same input, same answer, on any machine, forever.
bucket = stable_bucket(speaker_id)
split = "test" if bucket < 0.03 else "valid" if bucket < 0.06 else "train"

The split is now a pure function of the speaker id. There's no state to lose, no seed to mismatch, no file to drift. Months later I needed genuinely held-out audio for a load test; I imported the same function and got the same 3% instantly, with no archaeology.

Then assert it, rather than trusting the hash:

assert not (set(train_speakers) & set(test_speakers)), "speaker in two splits"

A speaker leaking across splits silently invalidates every number downstream. That is exactly the class of bug worth an assertion.

Treat the free tier as a design constraint

Free GPU tiers have session limits — a hard wall a few hours in, and your process is gone. Most people treat this as a reason not to use them. It's better treated as a spec.

What it forced:

• Checkpoint every 250 steps, keeping only the last and the best. Peak checkpoint disk stays bounded and a killed session costs minutes, not hours.
• Publish intermediate artifacts on a schedule, not just at the end. A run that dies at step 4,800 should still leave you something usable.
• Emit metrics as JSONL continuously, so the training curve survives even if the process doesn't.
• Validate config at build time. The trainer rejected validate_every_n_steps unless it was a multiple of the metrics-publish interval. Learning that from a crash five minutes into a six-hour run is a waste of a session, so the config generator now enforces it before submitting.

That last one generalizes: every constraint you learn from a crash should become an assertion in the generator. Otherwise you re-learn it, and the second time is more annoying than the first.

The result was a run I could lose at any moment without losing the work — which incidentally is also what you want from paid infrastructure. The free tier just makes you build it on day one.

Environment resolution is where the time actually goes

Two failures cost more debugging than everything model-related combined. Both are worth knowing.

A version specifier that silently downgraded. The library declared requires-python <=3.12. The machine ran Python 3.12.11. Under PEP 440, <=3.12 excludes 3.12.11, because 3.12.11 sorts after 3.12. So the package was ineligible, and pip did what pip does: it walked backwards until it found a release with looser constraints, installed a version from before the feature I needed existed, and reported success.

The failure surfaced as a missing submodule, which reads like a broken install rather than a wrong version. The fix was to install from a pinned source checkout with dependency resolution disabled — but the lesson is that "pip install succeeded" and "you have the version you asked for" are different claims. Assert the second one:

import importlib.util
assert importlib.util.find_spec("the_package"), "install reported success but the module is absent"

An editable install invisible to the running process. pip install -e writes a .pth file that Python processes at interpreter startup. Install it from inside an already-running notebook and the current process will never see it, no matter how many times you re-run the import cell. The install genuinely worked. The interpreter simply wasn't going to look again.

Both bugs share a shape: a tool reported success, and success meant something narrower than I assumed. In a notebook environment where you can't casually restart, that gap is expensive.

Run the A/B, even when you're confident

The fine-tuned model was obviously better. I nearly published against the original baseline and moved on.

I spent one more free session scoring the base model on the full test set instead, and the baseline moved: what had been published as 0.840 error / 70% correct-script was actually 0.808 / 75%. The original figure came from 200 clips on a single data shard.

So my headline improvement was a few points smaller than first claimed — an error in the direction that flattered me, which is the direction errors always seem to go. The published result is the corrected one.

The rule I'd extract: an A/B where the two sides were measured on different test sets is not an A/B. It's two numbers next to each other. Re-running the baseline cost one free session. Publishing an inflated number costs credibility the first time someone checks.

The scoreboard that mattered

Beyond the aggregate metrics, the harness reported a script-level breakdown:

devanagari      3839    100%
latin/other        0      0%
mixed              0      0%
empty              1      0%

baseline: 75% devanagari / 15% latin / 3% mixed / 6% empty

That table is where the actual insight lived. Word error rate said "bad, then better." The breakdown said "it was writing the language in the wrong alphabet, and now it isn't" — a completely different problem with a completely different fix. A single scalar would never have shown me that.

Build the per-failure-mode counter. It costs ten lines and it's usually where the finding is.

What actually made it repeatable

Stripped down:

1. Eval harness first, importing the shipping code.
2. Splits as a pure function of a stable id — no stored state, assertions on disjointness.
3. The session limit as a spec: frequent checkpoints, streamed metrics, intermediate publishing.
4. Config validated before submission, with every crash-learned rule encoded there.
5. Version assertions after install, because "succeeded" is not "correct."
6. Both sides of the A/B measured together, on the same set, with the same code.
7. A failure-mode breakdown next to the headline metric.

None of that is machine learning. All of it is why the machine learning was believable — and it's the same discipline that makes a production readiness review worth doing before you ship anything else.

The model was the easy part. It usually is.