Having a fine-tuned model is maybe a third of the work. The rest is turning a checkpoint into something an application can call without the application knowing or caring that a neural network is involved.
This is the full path a voice message takes, and the decisions along it that were not obvious to me before I made them the wrong way first.
Speak the protocol that already exists
The endpoint is deliberately shaped like OpenAI's transcription API:
POST /v1/audio/transcriptions (multipart: file, model, response_format, …)
GET /health
Not because that shape is elegant, but because every client library, every retry wrapper, and every developer's muscle memory already targets it. Migrating from a hosted API becomes a base-URL change instead of an integration project. Migrating back, if the self-hosted model disappoints, is the same one-line change — which is the property that makes the decision reversible and therefore easy to make.
That means accepting fields you ignore:
@app.post("/v1/audio/transcriptions")
async def transcribe(
file: UploadFile = File(...),
model: str = Form(default=MODEL_NAME),
response_format: str = Form(default="json"),
language: str | None = Form(default=None), # accepted, ignored
temperature: float = Form(default=0.0), # accepted, ignored
):
language, prompt and temperature are meaningless here — a CTC model decodes against one shared vocabulary, has no sampling temperature and no prompt channel. Rejecting them would be more "correct" and would break drop-in clients for zero benefit. Accept and ignore, and say so in a comment so the next reader doesn't think it's a bug.
Let ffmpeg do the decoding
Real voice notes arrive as Opus in an Ogg container, sometimes as m4a, occasionally as something surprising. Python audio libraries handle a subset well and the rest badly.
def decode_audio(data: bytes) -> np.ndarray:
"""Any container (opus/ogg/m4a/mp3/wav/webm) -> mono 16 kHz float32."""
proc = subprocess.run(
["ffmpeg", "-nostdin", "-loglevel", "error", "-i", "pipe:0",
"-f", "f32le", "-ac", "1", "-ar", "16000", "pipe:1"],
input=data, capture_output=True,
)
One subprocess, every format, no dependency matrix. It also gives you a clean failure boundary: if ffmpeg can't decode it, that's a 400, not a 500. Distinguishing "your file is broken" from "my service is broken" is worth building in from the start, because you will be reading those logs at some point.
The memory bug that decides your instance size
Inference memory was the thing that nearly killed the deployment, and the cause is worth knowing if you serve any ONNX model on CPU.
Steady-state memory tracked the longest audio clip ever processed and never came back down:
| longest chunk decoded | resident memory |
|---|---|
| 8s | 713 MB |
| 15s | ~950 MB |
| 30s | 1,373 MB |
ONNX Runtime's CPU memory arena grows to fit the largest allocation it has ever served and holds onto it. That's good for throughput — no repeated allocation churn — and it means one 30-second clip permanently sets your process's footprint, hours after that request finished.
Thread count barely mattered: going from 1 to 2 threads moved memory by 3 MB. The chunk length was everything.
So cap it, and split long audio at the quietest frame:
MAX_CHUNK_SECONDS = float(os.environ.get("STT_MAX_CHUNK_SECONDS", "10"))
The surprise: shorter chunks scored better. Word error rate on concatenated long-form audio went 0.639 unsplit to 0.534 at a 10-second cap, because the model was fine-tuned on short utterances and long context did it no favours. The memory fix improved accuracy. That is rare enough to be suspicious of, so I measured it twice.
One trap, since it cost me an afternoon:
# WRONG — binds the default at definition time. Reassigning the module
# global later does nothing, and the config knob silently has no effect.
def split_audio(samples, max_seconds=MAX_CHUNK_SECONDS): ...
Python evaluates default arguments once, when the function is defined. Making a constant configurable does nothing unless callers actually pass it. The recognizer now holds the value per instance, and there's a test that fails if that regresses — verified by reintroducing the bug and watching it go red.
The final memory model
After the cap, resident memory is predictable:
RSS ≈ 710 MB (arena, fixed by max chunk length)
+ 64 KB per second of clip (decoded PCM held before chunking)
A three-minute clip costs about 11 MB extra and gives it back. Clip length no longer raises the ceiling; only chunk length does. That's the difference between a service you can size and one you can only hope about — and it lets you plug real numbers into a throughput and concurrency calculation instead of guessing.
Make the container refuse to ship the wrong model
The base model and the fine-tuned model are the same architecture, the same file size, and both serve without complaint. They score wildly differently. Nothing in a running service tells you which one you have.
Two guards. First, the build fails rather than falling back:
ARG MODEL_SOURCE=local
RUN if [ ! -s /models/model.int8.onnx ]; then \
echo "model weights missing — refusing to build a service that would serve the wrong model"; \
exit 1; \
fi
Second, the running service reports what's baked into it:
def _model_name() -> str:
"""Written at build time, so it cannot drift from what's really served."""
with open("/models/MODEL_NAME") as fh:
return fh.read().strip()
GET /health returns that name. "Which model is in production?" is now a question with an answer instead of an inference from a container tag. Both guards exist because a silent fallback to the base model would look exactly like a working deploy.
Authentication, even on a private port
The service binds to the Docker bridge address, unreachable from outside the host. It still requires a bearer token:
def require_auth(creds = Depends(bearer)) -> None:
if not API_KEY:
return # unset = open, only safe on a private bind
if creds is None or creds.credentials != API_KEY:
raise HTTPException(401, "invalid or missing bearer token")
The token is self-generated (openssl rand -hex 32), not purchased. It costs nothing and it means the day someone publishes that port — during debugging, or in a config change six months from now — the failure is a 401 instead of an open transcription service on the internet.
The queue mistake worth avoiding
The application enqueues a background job per voice note. My first version put it on the shared ai queue alongside every other AI job.
That's wrong, and the reason is specific: transcription is CPU-bound on the same box and runs at 1.6x realtime, so a three-minute voice note holds a worker thread for nearly two minutes. Worse, the service decodes one clip at a time, so a second concurrent job doesn't do work — it sits on an open socket waiting its turn while still occupying a worker.
Three voice notes could occupy all three ai threads and make unrelated text customers wait for a reply. If the service ever wedged, that's the full client read timeout.
before outbound:8;ai:3;campaigns:2;scheduled:1;*:1
after outbound:8;ai:3;transcribe:1;campaigns:2;scheduled:1;*:1
One dedicated thread. One, deliberately — extra threads can't make the model decode faster, they'd only move the queue from the job runner into an HTTP wait. The general rule: a job that holds a thread while waiting should never share a pool with the job a user is actively waiting on.
Load-test with real audio, not a tone
My first load test generated sine tones with ffmpeg. Twelve requests, memory stayed under the cap, looked green.
It tested nothing. Tones decode to near-silence, so the recognizer's decode path exits early and the memory arena never grows the way speech grows it. Worse, it only watched the container — not whether the other services on the box still had room, which is the question that actually decides your instance size.
The real test: genuine speech, concatenated into message-shaped clips of 6 to 181 seconds, encoded as Opus at 24 kbps like a real voice note, using held-out speakers. 45 requests, 1,200 seconds of audio, up to 4 concurrent. Sampling the whole machine every 3 seconds — memory, swap counters, per-container usage — plus a latency probe against the co-tenant application.
memory cold 497 MB -> steady 711 MB, peak 720 MB
33 concurrent requests moved steady state by 2 MB
available never below 311 MB; under 5 MB paged
co-tenant app p50 5 ms, max 43 ms, 0 errors in 148 probes
0 failures, 0 empty transcripts, no restarts
That co-tenant line is the one that answered the actual question. Memory was never the constraint. The real limits turned out to be CPU credits on a burstable instance and queue latency — neither of which a bigger instance of the same family would fix.
One more distinction that cost me an hour: swap usage is not swap pressure. The box sat at 415 MB of swap the whole time and I nearly upgraded it over that. The number that matters is the rate — pages in and out per second, which was flat. Memory parked before the test is not memory being fought over during it.
The shape of the whole thing
gateway → API (bytes inline)
→ store to object storage, attach to the message row
→ enqueue transcription job (its own queue, 1 thread)
→ job: fetch bytes → POST to the local model service
→ model: ffmpeg decode → chunk at 10s → CTC decode → join
→ save transcript → trigger the debounced reply
Every arrow is a place it can fail, and every one degrades to the same fallback: no transcript, the assistant answers the placeholder, nobody is left unanswered. That was the design goal — not that transcription always works, but that its failure is never the customer's problem.
What I'd tell someone starting this
- Copy an existing API shape. It makes adoption and retreat equally cheap.
- Shell out to ffmpeg. Format handling is not where your effort pays.
- Find your memory ceiling and pin it to a knob, then measure whether the knob costs accuracy. Mine improved it.
- Make the artifact self-identifying. Build-time failure plus a
/healththat names the weights. - Isolate slow jobs into their own pool the moment they can starve a user-facing one.
- Load-test with real data on the real box, and watch the neighbours, not just the container.
None of this is exotic. It's the difference between a model that works in a notebook and a service that survives on a small production box next to everything else you're already running.