thai-asr-bench

How good is Thai speech-to-text, really? Nobody has measured it across the tools you'd actually use, so we did.

Thai is easy to transcribe badly and hard to benchmark. It writes without spaces between words, so word error rate needs a tokenizer just to decide where one word ends and the next starts — and every paper picks a different tokenizer, so the numbers don't compare. The commercial APIs publish English scores. The open leaderboards run open models against each other. No maintained benchmark runs Deepgram, Typhoon, and Whisper over the same Thai audio and shows you the result.

So nobody really knows how good Thai transcription is. This repo measures it.

What it measures

CER, not WER. Character error rate needs no tokenizer, so a model can't lower its score by picking a lenient one. One normalization pass sits between every model and its grade — NFC, lowercase, Thai digits to Arabic, spaces and punctuation removed — and it's the same pass for all of them. The full protocol is in METHODOLOGY.md.

Latin recall. Thai speakers drop English words mid-sentence — "แล้วก็ meeting ตอน 3 โมง." A single CER hides whether the model caught "meeting" or replaced it with something Thai-shaped. So we also report how many of the reference's English tokens made it into the transcript. It's blunt, but it catches what CER alone misses, and it's where a Thai-native model should pull ahead.

The board today

Three rows are real. The open Thai models still need a GPU run, and the table says so.

Model	Type	CER ↓	Latin recall ↑	n
Whisper large-v3	open	7.5%	50%	6
Deepgram Nova-3 (th)	commercial	9.7%	0%	6
Deepgram Nova-3 (multi)	commercial	99.3%	25%	6
Thonburian large-v3	open	needs GPU run	—	—
Typhoon ASR	open	needs GPU run	—	—

These are only six clips of FLEURS read speech, not conversation, so don't read too much into the exact numbers yet. Even so, the gaps are large. Deepgram's monolingual Thai model is close to Whisper on characters — 9.7% against 7.5% — but it drops every English word in the audio, so its Latin recall is 0%. Deepgram's multilingual mode, the one meant for code-switching, scores 99.3%: it has no Thai, so it romanizes the audio instead. That mode is the default many integrations use, including ours. Every number here reruns from this repo, which is why we shipped the harness before the leaderboard.

Run it

Fetch a public Thai eval set (FLEURS, CC-BY):

bun run fetch

Score every model you have credentials for. Missing keys become skipped rows, never guesses:

export WHISPER_CPP_MODEL=/path/to/ggml-large-v3.bin
bun run bench

The open Thai models want a GPU — on a laptop CPU, Thonburian runs at 0.08× realtime. So they run on Modal, the same hosted GPU compute pii-proxy trained on. Inference runs on an L4; the scoring runs locally, through the same scorer as everything else:

modal run scripts/modal_asr.py
bun run ingest

You now have results/results.md — the board, rebuilt from your own run.

What we won't do

• No invented numbers. An empty cell stays empty until a real run fills it.
• No private audio. Only publicly-licensed eval sets go in the repo. Real recordings people trusted us with never become benchmark data.
• No unreproducible rows. If you can't rerun it from this repo, it doesn't get a line.

We publish the score even when it embarrasses us. Building this is how we found that our own Deepgram integration defaulted to multi — the mode that scores 99.3% here because it has no Thai — so Thai audio was quietly romanized into gibberish before it reached a transcript.

Contenders

Model	Kind	How it runs
Whisper large-v3	open baseline	local, whisper.cpp
Thonburian large-v3	Thai-tuned Whisper	Modal L4, faster-whisper
Typhoon ASR	Thai-native (SCB 10X)	hosted API / NeMo
Deepgram Nova-3 (th)	commercial	API, monolingual Thai
Deepgram Nova-3 (multi)	commercial	API, code-switch mode, which has no Thai

Adding one is a file in adapters/ that turns audio into text; the scorer takes it from there.

---

Part of Daslab's open benchmarks. MIT licensed.