The complete ML pipeline: from raw data to production, with the calculations

Most ML tutorials teach you to train a model. Almost none teach you the other 90% — how to know your data is right, how to split it so your results mean something, which knobs actually matter and how to compute them, and everything that happens between "the notebook printed a good number" and "real users are hitting it".

This is that walkthrough. Every stage, in order, with the arithmetic. The examples come from a real speech recognition model I fine-tuned and shipped on free GPU quota, so the numbers are ones I actually had to defend rather than round figures invented for a blog.

The map:

data → splits → targets → metrics → baseline → training
     → evaluation → packaging → serving → deployment → monitoring

You can read it end to end, or jump to the stage you're stuck on.

Stage 1 — Data, and what "clean" actually means

Everyone says garbage in, garbage out. Nobody says what garbage looks like. Concretely, it's four things:

1. Wrong or inconsistent labels. The corpus I used transcribed the numeral 007 three different ways across otherwise identical audio. That's not noise you can average out — it's contradictory targets, where the same sound maps to different answers. The model can only learn the average of a contradiction, which is worse than either answer. Filtering every utterance containing a numeral dropped 6.2% of the corpus and improved the result.

2. Characters your model cannot produce. More on this in Stage 3; it silently cost me a nine-hour training run.

3. Duration outliers. Clips shorter than your model's receptive field produce targets longer than the input can encode — mathematically unlearnable. Clips longer than what you'll ever serve teach a shape you never see. I filtered to 0.5–30 seconds, where 30 s matched the serving limit exactly.

4. Licensing you can't actually use. Not a modelling issue, a shipping issue. A dataset with no declared licence is worse than a restrictive one, because "no licence" means no permission by default. Check before you train, not after.

The rule that matters most: verify contents, not descriptions. I found a dataset whose card described English speech. 630 of its 642 rows were a different language entirely. A head -20 would have caught it, and nearly didn't get run.

# The cheapest possible data check, and it finds real problems
import collections, re
scripts = collections.Counter(
    "deva" if re.search(r"[ऀ-ॿ]", r["text"]) else "latin"
    for r in rows
)
print(scripts)   # Counter({'deva': 630, 'latin': 12})  <- card said English

Stage 2 — Splits, and how a random split lies to you

You split data into train (the model learns from it), validation (you watch it during training to catch overfitting), and test (touched once, at the end, to report a number).

The naive approach is a random split. For most real data, a random split gives you an inflated result — because it leaks.

Leakage is when information from your test set reaches training. It doesn't need to be the same rows. If one speaker contributes 40 clips and you split randomly, roughly 32 land in train and 8 in test — so at test time the model is recognising a voice it has already memorised, not generalising to a new one. Your reported error is optimistic, sometimes by a lot, and you find out in production.

The fix is to split along the axis you're claiming to generalise over:

# Speaker-disjoint split. Deterministic hash, NOT random.shuffle —
# the split must be identical on every machine and every re-run, or
# "test" quietly becomes training data next time someone regenerates it.
def bucket(speaker_id: str, buckets: int = 100) -> int:
    return int(hashlib.sha256(speaker_id.encode()).hexdigest(), 16) % buckets

test  = [r for r in rows if bucket(r.speaker) < 5]        # 5%
valid = [r for r in rows if 5 <= bucket(r.speaker) < 10]  # 5%
train = [r for r in rows if bucket(r.speaker) >= 10]      # 90%

Two properties worth naming. It's deterministic — same input, same split, forever, so a regenerated manifest can't silently move a test speaker into training. And it's stable under growth — adding new speakers doesn't reshuffle existing ones, so today's numbers stay comparable to last month's.

Later in this project I used the same idea for a stronger claim. I wanted to argue the model generalised to an accent it had never heard, so I held out every speaker of that accent and trained on eighteen others. That turns a soft claim into a measurable one. Choose your split to match the sentence you want to be able to say.

Stage 3 — Targets and the vocabulary trap

Your model can only emit symbols in its vocabulary (or tokeniser) — the fixed list of characters or sub-words it knows. Anything outside it is unrepresentable.

This sounds obvious and it is a very common, very expensive bug.

The tokeniser I was using had lowercase Latin, one non-Latin script, and exactly two punctuation marks: ' and -. No uppercase. My English training data was ordinary written English — capitals, commas, full stops.

85% of my training targets contained characters the model physically could not emit.

It learned to want them anyway, and produced the unknown-token symbol where they belonged:

"That"     ->  <unk>hat
"English"  ->  <unk>nglish

That cost about 0.20 WER and half the character error, for reasons that have nothing to do with speech. Worse, it degraded the acoustic learning itself — after fixing it, the score beat what I'd predicted from simply deleting the bad tokens from the old output.

The fix is fifteen lines, and the guard is more important than the fix:

def load_vocab(tokens_path: str) -> set[str]:
    # One symbol per line: the exact set the model can emit.
    return {line.split()[0] for line in open(tokens_path) if line.strip()}

def normalise(text: str, vocab: set[str]) -> str:
    text = text.lower()
    return "".join(c for c in text if c in vocab or c.isspace())

def assert_emittable(rows, vocab):
    """Fail the DATA PREP, not the training run, on an unemittable target."""
    bad = {c for r in rows for c in r["text"] if not c.isspace() and c not in vocab}
    if bad:
        raise SystemExit(f"{len(bad)} characters outside the vocabulary: {sorted(bad)[:20]}")

Run the assertion every time you add a corpus. It is one file read. It stands between you and a multi-hour run that produces a plausible-looking bad model.

Stage 4 — Metrics, and when they lie

The formulas

Word Error Rate is edit distance at the word level:

WER = (S + D + I) / N

S = substitutions   (wrong word)
D = deletions       (missing word)
I = insertions      (invented word)
N = number of words in the reference

Worked example. Reference the cat sat on the mat (N = 6), hypothesis the cat sat on mat:

S = 0, D = 1 ("the"), I = 0
WER = (0 + 1 + 0) / 6 = 0.167

Character Error Rate is the same formula over characters. It's more forgiving and more informative for morphologically rich languages, where one wrong suffix fails a whole word under WER but barely moves CER.

Note WER can exceed 1.0. If the model hallucinates, insertions are unbounded while N is fixed. Seeing WER 1.013 means it emitted more wrong words than the reference contained.

The trap: one number, two different failures

My model produced WER 1.0 in two completely different situations:

1. It misheard the audio entirely.
2. It heard perfectly and wrote in the wrong alphabet — phonetically correct, every word "wrong".

Same metric value. Opposite fixes. Case 1 needs acoustic data; case 2 needs output-vocabulary repair. A metric that cannot distinguish your failure modes is not measuring your bug.

So add a task-specific metric. Mine was ten lines:

def script_of(text: str) -> str:
    deva = len(re.findall(r"[ऀ-ॣ॰-ॿ]", text))
    latin = len(re.findall(r"[A-Za-z]", text))
    if deva == latin:
        return ""          # tie or empty: NO opinion, never a silent pass
    return "deva" if deva > latin else "latin"

That turned an unactionable "WER is 1.0" into "script accuracy is 0.000, so the encoder is fine and the output space is broken" — which is a different project.

Two subtleties that generalise beyond scripts. Digits shouldn't vote: the Devanagari Unicode block contains its own digits, and २०२६ and 2026 are the same year, so counting them as evidence of language is wrong. And abstain instead of guessing: an empty output has no script, and scoring it as correct would hide the exact failure the metric exists to find.

Stage 5 — Baseline before you train

Measure the un-trained, off-the-shelf model on your own test set first. Always. It costs an hour and it buys you:

• A number to beat. Without it, "WER 0.42" is meaningless.
• A working harness. You'll debug your evaluation code on the baseline instead of discovering it's broken after a nine-hour run.
• A go/no-go. Sometimes the stock model is already good enough and the correct engineering decision is to not train anything.

Critically, score the baseline with the same code that will serve traffic. A benchmark that measures different code than it ships is worse than no benchmark, because it produces confident wrong numbers. I run evaluation through the same inference class the server uses.

Stage 6 — Training: the knobs that matter, and how to compute them

Batch size, gradient accumulation, effective batch

GPU memory limits how many samples you can process at once. Gradient accumulation lets you simulate a bigger batch by summing gradients over several forward passes before updating weights:

effective_batch = per_step_batch × grad_accum_steps × num_gpus

The reference recipe I started from assumed 32 A100s. I had one 16GB T4. I cut the per-step batch by 8× to fit in memory and set grad_accum = 8 to recover a comparable effective batch. Same maths, 1/32 the hardware, roughly the same optimisation behaviour.

Steps vs epochs — the calculation people skip

An epoch is one full pass over your data. A step is one weight update. Recipes are usually written in steps, and you should always convert to epochs to sanity-check:

audio_per_step = max_elements_per_batch × grad_accum
               = 60 s × 8 = 480 s ≈ 240 s of usable audio after padding

steps_per_epoch = corpus_seconds / audio_per_step
                = (91 h × 3600) / 240
                = 327,600 / 240
                ≈ 1,365 steps

epochs = num_steps / steps_per_epoch
       = 5,000 / 1,365
       ≈ 3.7 epochs

3.7 epochs is a sensible fine-tune. Had I taken the reference recipe's step count literally, it would have implied roughly 540 epochs on a corpus this size — a guaranteed overfit and days of wasted quota. This one calculation is the highest-value five minutes in the whole process.

Learning rate

Fine-tuning uses a much smaller learning rate than training from scratch — you're nudging existing weights, not discovering them. 1e-5 is a common fine-tuning value; from-scratch training is often 1e-3. Too high and you get catastrophic forgetting: the model overwrites what it already knew. Too low and nothing moves.

Precision: fp32, fp16, bf16

Lower precision means less memory and faster maths.

This is hardware-gated and it will bite you. bf16 requires compute capability 8.0+. On a T4 (7.5) the reference config's bfloat16 simply fails; on an older P100 (6.0) fp16 has no tensor cores and can produce NaN losses with no speedup. Check your GPU's compute capability before trusting a config someone else wrote.

cc = torch.cuda.get_device_capability(0)
precision = "bfloat16" if cc[0] >= 8 else "float16"

Mixing datasets: sampling temperature

When you train on several corpora or languages of very different sizes, proportional sampling drowns the small one. The standard control is a temperature β:

p_i ∝ n_i^β

β = 1.0  → proportional to size
β = 0.5  → proportional to √size  (partial balancing)
β = 0.0  → uniform, regardless of size

My mixture was 91 h of one language and 8.26 h of another:

β = 1.0:  91 / (91 + 8.26)          = 92% / 8%
β = 0.5:  √91 / (√91 + √8.26)
        = 9.54 / (9.54 + 2.87)      = 77% / 23%
β = 0.0:  50% / 50%

At 8% the small corpus couldn't move anything. At 50% it would swamp the large one. I used β = 0.5 for 23%, which worked — but note this knob trades one against the other. It cannot raise both. If you need both to improve, you need more data, not a different β. I learned that the hard way (Stage 7).

Freezing layers

You can hold parts of the network fixed. Freezing the encoder trains only the output head — cheaper and safer against forgetting. But if the encoder is what's wrong, freezing it guarantees failure. In my case the acoustic front-end was mis-routing, so I froze nothing. Decide from the diagnosis, not from habit.

Stage 7 — Evaluation design, where most conclusions go wrong

Compare on identical inputs

I had two numbers: the single-language model at 0.379 and the mixture at 0.418. That looks like a small regression. It's actually not a comparison at all — the first was measured on 200 clips, the second on 3,840. Different test sets produce different numbers for reasons unrelated to the model.

So I re-scored both models on 1,000 byte-identical clips:

model       WER      CER
v1        0.392    0.195
v3        0.429    0.214
delta:   +0.037   (v3 worse)

The regression was real, and roughly 9% relative — but I could only make that claim after an A/B on identical inputs. Two carefully-measured numbers from two different test sets cannot be subtracted. Ever.

Note also that the comparison I had been making — mixture-run-A vs mixture-run-B — could never have detected the cost, because both contained English. Make sure your control actually differs in the one variable you're testing.

How many samples do you need?

Full test sets are often overkill for a comparison. Standard error of a proportion:

SE ≈ √(p(1−p)/n)

n = 1,000, p ≈ 0.4:  SE = √(0.4 × 0.6 / 1000) ≈ 0.0155
95% CI ≈ ±1.96 × SE ≈ ±0.030

So with 1,000 clips I can resolve differences of roughly 3 points. My observed delta was 3.7 points — detectable, but only just. If I'd needed to resolve 1 point I'd have needed roughly:

n ≈ (1.96 / 0.01)² × 0.4 × 0.6 ≈ 9,220 clips

Sample size is a choice about what you need to detect, not a number you inherit. Scoring 1,000 instead of 3,840 turned a multi-hour job into twenty minutes, and I stated the limit rather than pretending to precision I didn't have.

Watch for the unrepresentative test set

My original English testing used clean, native-accent audio. It passed. Every real user is a non-native speaker, and for them the model failed 100% of the time. The sample wasn't too small — it was drawn from the wrong distribution. Ask constantly: does my test set look like my traffic?

Stage 8 — Packaging: quantisation and export

A trained checkpoint is not a deployable artifact. Two transformations get you there.

Export converts framework-specific weights into a portable graph — ONNX is the common target. It lets you serve without the training framework installed, which typically takes a multi-GB dependency tree down to a small runtime.

Quantisation reduces numeric precision of the weights:

fp32 → int8  = 4× smaller, 4× less memory bandwidth

my model:  1,242 MB (fp32)  →  348 MB (int8)

Accuracy cost is usually small (often under 1% relative) but it is not zero, and you must measure it rather than assume. Score the quantised artifact, not the checkpoint — that's what users will hit.

Stage 9 — Serving capacity, with the arithmetic

Real-Time Factor

For streaming/audio models, RTF is the core throughput number:

RTF = processing_time / audio_duration

RTF < 1  → faster than real time
RTF = 0.168 → 6× real time (my model, one CPU thread)

One worker at RTF 0.168 can sustain about 6 seconds of audio per second of wall clock.

Little's Law

The most useful queueing result in production engineering:

L = λ × W

L = items in the system
λ = arrival rate
W = average time in the system

Say voice notes arrive at 0.5/second, averaging 8 seconds of audio each, with RTF 0.168 and one worker:

service time  W = 8 × 0.168 = 1.34 s
capacity      μ = 1 / 1.34 ≈ 0.74 requests/s
utilisation   ρ = λ / μ = 0.5 / 0.744 ≈ 0.67

At 67% utilisation you're fine. But queue wait explodes non-linearly as ρ → 1:

M/M/1 average wait:  W_q = ρ / (μ − λ)

ρ = 0.67:  W_q = 0.67 / (0.744 − 0.50) ≈ 2.7 s
ρ = 0.90:  W_q = 0.90 / (0.744 − 0.67) ≈ 12.2 s
ρ = 0.95:  W_q = 0.95 / (0.744 − 0.71) ≈ 28 s

Plan for 70% peak utilisation, not 95%. The last 25% of capacity costs you an order of magnitude in latency. Plug your own numbers into the Throughput & Concurrency Calculator and the Latency Budget Calculator.

Memory behaviour you won't predict from the model size

My 348 MB model used ~709 MB resident. The gap is the inference runtime's memory arena: it grows to the largest input it has ever processed and never gives it back. So peak memory is set by your longest ever input, not your average.

That's a lever, not just a fact. Capping chunk length at 10 seconds took resident memory from 1,373 MB to 737 MB — and scored better, because the model was trained on short utterances. Measure resident memory under realistic load; don't extrapolate from file size.

Then set a hard container memory limit. Without one, a spike gets a victim chosen by the kernel's OOM killer — usually your database. With one, only the model container dies and restarts, and the blast radius is a single failed request.

Stage 10 — Deployment and verifying the artifact

The question "is the new model actually live?" has answers of increasing strength, and only the last one is real:

1. The workflow was green. Proves a pipeline ran. Not that it did what you think.
2. The health endpoint reports a version string. Build-time metadata — proves which image, not which weights.
3. The container's image digest and architecture. Rules out a stale mutable tag and a wrong-platform pull.
4. A checksum of the model file inside the running container, compared to the file training produced.

docker exec <container> md5sum /models/model.int8.onnx
# 3725b3929dd32585f0e0c7d5aa0f289b   <- matches the training output
# ddda2e78ba003edea348017b42862bf3   <- the previous model

Byte-level identity, ten seconds, no ambiguity. Do that one.

Three deployment details that regularly cause outages:

• Mutable tags. :latest means "whatever was pushed most recently", so what you tested and what deploys can differ. Pin digests, or verify after every deploy.
• Architecture. ARM and x86 images are not interchangeable. Build multi-arch, and check uname -m on the target rather than trusting a comment in a config file. A stale comment cost me a cancelled build.
• Sidecars aren't restarted by app deploys. Many orchestrators treat auxiliary containers separately, so a deploy leaves the old model running indefinitely. Worse, if the deploy enables the feature flag before the sidecar exists, you get a window of connection failures. Sequence deliberately.

Stage 11 — Monitoring, and the failure that leaves no trace

The bug that cost me the most debugging time produced no error, no failed job, and no log line. A request path returned early on an empty result. From the outside it was indistinguishable from working correctly and simply having nothing to say.

# Before — a silent early return
if not text:
    return

# After
if not text:
    logger.warning(
        "no output for request %s: model returned nothing usable (%s)",
        request_id, error or "empty result",
    )
    return

Every silent early return in a pipeline is a future debugging session. Log the reason, the identifier, and enough context to distinguish "correctly had nothing to say" from "broke and swallowed it".

Beyond that, track: input distribution drift (are clips getting shorter?), latency percentiles (p50 and p99, not the mean), error and fallback rates, and resource ceilings. And keep a small canary set of real production inputs you re-score after every model change — that's what catches the regression your benchmark can't see.

Glossary

The checklist

Data and splits:

• Inspected actual rows, not the dataset description
• Licence verified before training
• Filtered contradictory labels and duration outliers
• Split along the axis you claim to generalise over
• Split is deterministic and stable as data grows

Targets and metrics:

• Every training target validated against the tokeniser's vocabulary
• A metric that distinguishes your failure modes, not just aggregate error
• Baseline measured, using the same code that will serve traffic

Training:

• Steps converted to epochs and sanity-checked
• Precision matched to your GPU's compute capability
• Mixture weights computed, not guessed

Evaluation:

• A/B on byte-identical inputs
• Sample size justified against the difference you need to detect
• Test set resembles production traffic

Ship:

• Quantised artifact scored, not just the checkpoint
• Capacity computed at ~70% peak utilisation
• Resident memory measured under realistic load, with a hard container limit
• Deployed artifact verified by checksum
• Architecture verified on the target host
• Silent early-return paths log a reason
• Canary set re-scored after every model change

The thing worth remembering

Almost every expensive mistake in this list produced a green signal while something underneath was wrong. The workflow passed. The metric looked fine. The dataset card said English. The comparison showed no regression.

Machine learning in production is mostly not a modelling discipline. It's a measurement discipline — building instruments you can trust, then trusting them over your intuition. The modelling is the easy part; knowing whether it worked is the job.

Before you ship, run your service through the Production Readiness Scorecard — it covers the operational half of this list.