case_study
Self-hosted speech-to-text
Speech recognition engine for a low-resource language
Solo — model fine-tuning, inference service, and deployment
- Python
- PyTorch
- ONNX Runtime
- FastAPI
- Docker
problem
The problem
A large share of the messages arriving at the product were voice notes, and every one of them was being dropped. The assistant received the text "[voice message]" and answered that, so it was replying to the fact that someone had spoken rather than to anything they said.
The obvious fixes did not survive contact with the requirements. Commercial speech APIs bill per minute of audio and would have sent customers' recordings to a third party. The strongest open-source alternative could not write the language at all without fine-tuning — measured zero-shot word error rates were above 100%, because it transcribed into the wrong alphabet.
The deeper issue was diagnostic rather than financial. The best available multilingual model was not failing to hear the language. It heard it and then wrote it in Latin letters 15% of the time, because the architecture takes no language hint and nothing told it which script to use. That is a different problem from inaccuracy, and it needs a different fix.
approach
What I engineered
- Fine-tuned an open-source multilingual CTC speech model on ~96 hours of public read speech, on a free-tier cloud GPU — 5,000 steps in 5.4 hours, at no cost, with checkpointing designed around the free tier's session limit.
- Built the evaluation harness first and ran the true A/B through it: base and fine-tuned models scored on the same 3,840 speaker-disjoint held-out utterances, using the same recognizer class that serves production, so the benchmark measures the code that actually ships.
- Made the train/validation/test split speaker-disjoint via a deterministic hash of the speaker id, so the split is reproducible without storing a seed and no voice appears in two splits.
- Wrote an OpenAI-compatible transcription API around the exported int8 ONNX model, so any existing client migrates by changing a base URL, and packaged it as a container that fails the build rather than silently shipping the wrong weights.
- Diagnosed and fixed a memory ceiling that blocked deployment: the ONNX Runtime CPU arena grows to the longest audio chunk ever decoded and never releases it. Capping chunk length took steady-state memory from 1,373 MB to roughly 710 MB — and shorter chunks scored better, so the fix cost no accuracy.
- Load-tested on the target hardware with real speech shaped like real messages, sampling the whole machine rather than the container, then isolated transcription onto its own worker queue so a long clip could not delay unrelated replies.
result
The result
Word error rate fell from 0.808 to 0.379 and character error rate from 0.365 to 0.186, on identical held-out audio. The wrong-alphabet failure went from 15% of utterances to zero, and correct-script output from 75% to 100%. Empty transcriptions fell from 248 utterances to one.
The service runs on hardware that was already paid for, with no per-minute billing and no customer audio leaving the machine. Under load it holds a flat memory footprint and transcribes at 1.6x realtime, and the load test proved the co-tenant application kept serving at a 5 ms median throughout.
The engagement also produced a documented negative result: freezing the model's encoder — the textbook fix for the multilingual regression fine-tuning caused — was measured and rejected at word error rate 0.659, roughly 40% of the gain. It is written up rather than quietly dropped, because knowing which principled fix does not work is worth as much as the one that does.
- 0.808 → 0.379
- word error rate, same held-out set
- 75% → 100%
- output in the correct script
- $0
- training cost · free-tier GPU
My own measurements, taken on 3,840 speaker-disjoint held-out utterances from a public speech corpus using the shipped inference code, 2026. Client identifiers, product details, and infrastructure specifics are deliberately omitted.
work_with_me
Have a project like this?
Tell me the problem on a 20-minute call and I'll tell you how I'd approach it, the same way Self-hosted speech-to-text started.
Discuss your project