Language models consumed 98.5% of analysis time when added to a spell checker

A hypothesis that is hard to argue with

Someone types kanus. No such word exists in Indonesian, so the spell checker has to guess what was meant. And here the trouble starts: more than one answer is equally plausible if all you look at is the shape of the letters. kamus, meaning dictionary, is one letter away. So is kaus, meaning t-shirt.

Only the sentence can settle it:

Saya pinjam kanus di perpustakaan.   ->  almost certainly "kamus" (dictionary)

Saya pakai kanus warna biru.         ->  almost certainly "kaus" (t-shirt)

The Indonesian correction engine we are building handles typos in two steps, and the second step was born from exactly that observation. The first step sweeps the lexicon for the word closest in form, using edit distance, then hands over a list of candidates. It is entirely blind to meaning. The second step reorders that list with a language model, in the hope that sentence context finishes the job.

The model we used is IndoBERT, a language model trained specifically on Indonesian text. It does one thing, and that happens to be the thing needed here: given a sentence with one word covered up, it judges which word most likely belongs in that slot. We run it through ONNX, a format that lets a model like this execute directly inside our engine without dragging Python along into production.

On paper the pair complements each other neatly. Edit distance knows shape, the language model knows meaning. What we measured was one thing: does the guess that reaches the user get right more often.

One switch, an answer key fixed in advance

Two arms, an identical pipeline, the only difference a switch turning the second step on or off.

The answer key is not set by human judgement. We corrupted 1,500 correct words under controlled conditions with a fixed seed, so the right answer was known before the engine saw the data. That matters because the party grading here is the party that built the engine, and without a key locked in advance, such grading has a predictable direction of bias.

11.2 points

The worst damage lands on the transposition class, down 28 points. That is the most purely mechanical class of all: kamsu from kamus, two letters swapped. Nothing there requires understanding meaning.

The column that pins down the cause

A large number alone does not identify the culprit. What makes this finding impossible to pin on another component is the following pair of columns.

The first compares the accuracy of symspell's raw top candidate against the accuracy of what was actually sent to the user:

The raw figure is identical in both arms. The raw material is the same, so the only thing that changed is what got chosen to send.

The second separates two causes of failure. Failing because the correct word never entered the candidate list, or failing because it entered but lost on ranking:

Retrieval matches to the unit. Ranking failures drop by 160 cases the moment the model is switched off.

What did not move at all, and why that matters

Reranking did not break everything. That is exactly what makes the finding credible, because a change that breaks everything usually means the measurement is wrong.

The real-text audit is identical right down to the per-class breakdown: merges 196, PUEBI 524, slang 290, splits 72, typos 1,401.

The reason is structural. Reranking only chooses among existing candidates; it never decides whether something gets flagged. That was settled by the first step. So the entire cost lands in one place: which answer gets sent.

What it cost in time

Measured over 10,000 news sentences, 148,839 words.

Reranking consumed 98.5 percent of analysis time. Switching it off made the engine 67 times faster, with an identical number of findings.

98.5 percent of our time budget was spent picking answers that were wrong more often.

Why it was not caught sooner

This section is the most useful part for anyone else, because the mistake had nothing to do with Indonesian and nothing to do with the model.

We had a monitoring tool, and it reported: reranking changes the top answer in 48.2 percent of cases, but 99 percent of those moves are between candidates at equal edit distance, so they are legitimate. Conclusion: safe.

That conclusion was wrong, and wrong in a tidy way. The tool had no answer key. It could only see how often the ranking changed, not whether the change was right. And all of the damage occurred precisely where it had always been considered safest: among candidates at equal edit distance.

The rule: a metric that measures how active a component is often gets mistaken for a metric that measures how useful it is. Without an answer key the two cannot be told apart, and the first is far easier to build.

This was the second measurement, not the first

Seven months earlier, a separate internal measurement recorded raw symspell as correct 70.7 percent of the time in first position, falling to 66.3 percent after reranking. Down 4.4 points, the same direction.

That finding was written as a comment inside the source code, and never acted on.

So the first piece of evidence had been sitting in our own repository for more than half a year, in a form nobody would ever read. Two independent measurements, months apart, different methods, same direction. What is different this time is not the number but that it landed somewhere with consequences.

What must not be concluded

Not that language models are useless for text correction.

Our probe damages letters mechanically, which is home turf for edit distance. The error class that contextual models actually promise to solve, a correctly spelled word chosen wrongly for the meaning of the sentence, is not represented at all in this test. For that class we have no evidence in either direction.

What is proven is narrower and firmer: on mechanical typos, the shape of a word is stronger evidence than the plausibility of its meaning, and letting meaning override shape is costly. A typo is a failure of the finger, not of thought, so it makes sense that the thing that knows the answer best is similarity of form.

The switch was deliberately kept in the code rather than deleted. If we ever get a real corpus of drafts with contextual errors, this question can be measured again without touching a single line.