In a lot of the world, people don't type. They hold the button and talk. Voice notes are the default way to send a message, especially where typing means switching keyboards or spelling a language whose script your phone barely supports.
Which makes it awkward that most AI assistants quietly ignore them.
Not "handle them badly" — ignore them. A voice note arrives, the system can't turn it into text, and the assistant receives something like [voice message] instead of words. It then answers that. The customer said something specific and got a reply to the fact that they spoke. From the outside it looks like the AI wasn't listening, because it wasn't.
I recently fixed this for a product where a large share of incoming messages were voice notes. The interesting part wasn't the fix. It was the diagnosis, which turned out to be the opposite of what everyone assumes.
The assumption: the model can't understand the language
Speech recognition is overwhelmingly built, benchmarked, and tuned for English. Languages with fewer speakers online — often called low-resource languages, though there is nothing low-resource about a language 30 million people speak — get whatever generalization falls out of a multilingual model.
So the obvious theory is: the model doesn't know enough Nepali. And there's evidence for it. I measured a well-known open-source speech model on Nepali audio with no fine-tuning:
| Model size | Word error rate |
|---|---|
| tiny | 101.8% |
| base | 102.4% |
| small | 69.5% |
| medium | 54.4% |
Above 100% error looks like nonsense, and it's worth understanding why it happens: word error rate counts insertions as well as substitutions and deletions, so a model that outputs more wrong words than the reference had can score over 100%. It means the output isn't merely inaccurate. It's unusable.
The obvious conclusion is "the small models can't do this language, pay for a bigger one." That conclusion is wrong, and it's expensive to act on.
What was actually happening
Switching to a genuinely multilingual model — Meta's Omnilingual ASR, an Apache-2.0 release covering 1,600+ languages — the numbers improved but stayed bad: word error rate 0.808, meaning roughly four out of five words wrong.
Then I looked at the actual output instead of the metric.
The model was writing Nepali in Latin letters. Not gibberish — recognizable, phonetically sensible Nepali, spelled out in the English alphabet, like writing "namaste" instead of नमस्ते. On 15% of clips it did this. On another 3% it mixed the two alphabets in a single sentence.
Word error rate treats that as a total failure, and by its own rules it is: not one word matches. But it isn't a hearing problem. The model heard correctly and then made a writing decision, and the writing decision was wrong.
Once you see it that way, the cause is easy to find. That model family takes no language hint. You cannot tell it "this is Nepali." It gets audio, and it picks a script from one shared vocabulary covering every language it knows. Faced with sounds that exist in many languages, it sometimes picks the alphabet that dominates its training data.
This is the whole reason a bigger model was never going to fix it. Scale doesn't tell a model which alphabet you wanted. Nothing in the architecture was ever going to.
The fix, and why it's cheap
If the problem is "the model doesn't know which script to write in," the fix is to show it — a few thousand examples of this language written correctly.
That's fine-tuning, and its reputation for being expensive is mostly out of date for a job this narrow. The whole run:
- ~96 hours of freely available public read speech (OpenSLR-54, the open Nepali corpus)
- 5,000 training steps, 5.4 hours of GPU time
- A free-tier cloud GPU, the kind with a session timeout and no bill
- $0
Results, measured on 3,840 held-out clips from speakers the model never trained on:
| before | after | |
|---|---|---|
| Word error rate | 0.808 | 0.379 |
| Character error rate | 0.365 | 0.186 |
| Correct script | 75% | 100% |
| Wrong alphabet | 15% | 0 |
| Mixed alphabets | 3% | 0 |
| Heard nothing at all | 248 clips | 1 clip |
Errors more than halved. But the row that changed the product is "correct script." The wrong-alphabet failure — the one that made output unusable rather than merely imperfect — is gone.
The part nobody puts in the announcement
Fine-tuning is not free. It is a trade, and I paid for it.
Teaching the model Nepali cost it other languages. German and Spanish now come back transliterated into Nepali's alphabet — the exact mirror of the bug I'd just fixed. English and French survived, degraded but readable.
I found this by chance, after deploying, because someone tried a German clip. It should not have been chance, and the fix for that is a test rather than better luck: a script that runs the same clips through both models and fails if a non-Nepali clip comes back in the wrong script. That check now gates deployment. The full postmortem is here, including the principled fix I tried next and measured as worse.
For this product the trade is correct: the users speak Nepali and English, and English survived. For a general-purpose transcription service it would be the wrong call. The honest framing is that I built a specialist, not an upgrade.
Why self-hosted, not an API
With a working model there were two ways to ship it: rent transcription from a cloud provider, or run it myself.
Three things decided it.
Cost shape. Speech APIs bill per minute of audio. Voice notes are the cheapest message for a customer to send and the most expensive for you to process, so the bill scales with exactly the behaviour you're encouraging. A self-hosted model runs on hardware you already pay for, and the marginal cost of one more voice note is some CPU time.
Privacy that's structural. Customer voice notes are personal data. Not sending them anywhere is a stronger guarantee than a policy saying they won't be misused. "We don't retain your audio" is a promise; "the audio never left the server" is an architecture.
Coverage. For English, the commercial APIs are excellent and I would just buy one. For this language, they were the weaker option — which inverts the usual build-versus-buy instinct.
The result runs on the existing server, adds no per-message cost, transcribes about 1.6x faster than realtime, and holds a flat memory footprint under sustained load. Getting to flat was its own fight — the memory bug is in the postmortem — and it's the difference between a model that works on a laptop and a service that survives on a small production box.
What to take from this
If your product ignores voice messages, the reason is probably not that the technology isn't ready.
- Look at the output, not just the metric. A single number told me the model was terrible at Nepali. The transcripts told me it was good at Nepali and bad at choosing an alphabet. Those need opposite fixes, and only one of them is cheap.
- Check whether the failure is capability or configuration. Scaling up is the expensive answer, so it's worth ruling out first, not last.
- Fine-tuning a narrow, well-defined task is now genuinely accessible. Half a day of free GPU time closed the gap here.
- Every fine-tune is a trade. Write down what you gave up and put a test around it, or you'll find out from a user.
The broader point: "AI doesn't work well in my language" is often really "nobody has spent half a day pointing it at my language." That's a much smaller problem than it sounds, and it's one worth solving — for the many people whose first instinct is to hold the button and talk.