Two identically priced AI models tested on Indonesian benchmark: 56.52% vs 26.09%
Two AI models from two different companies, sold at the exact same price: $0.05 per million input tokens, $0.08 per million output tokens. Both equally fast. We gave both the same 69 items, and the results were more than two times apart. What we did not expect was that some of the errors were in our own answer key.
Key numbers
Where these come fromMeasured from runs-2026-08-06-idN-dua-model-murah-summary.md and 1 more
Same budget, double the result
If you are picking a cheap AI model to process Indonesian text, you usually compare price first, then speed. These two models defeat that approach, because they are identical on both counts.
| Mistral Small 3 | Llama 3.1 8B Instruct | |
|---|---|---|
| Company | Mistral AI | Meta |
| Input price / million tokens | $0.05 | $0.05 |
| Output price / million tokens | $0.08 | $0.08 |
| Latency ranking in catalog | #7 | #10 |
| Context window | 32,768 tokens | 131,072 tokens |
The price is exactly the same. Catalog speed is nearly identical. The one column that is clearly different actually favors Llama: its context window is four times larger.
So we gave both the same job.
The task: reading what people type in a hurry
The item set contains sentences written the way people type quickly on their phones, and the model's job is to return them to standard Indonesian.
source : Bsk aku blm bisa dtg, ada urusan.
should be : Besok aku belum bisa datang, ada urusan.
69 items, all free-form and not multiple choice, three repeated runs per item, temperature 0. Scoring used edit-based scoring: what is compared is the edit the model made to the source sentence, not the text character by character, so formatting differences that do not change meaning are not penalized.
| Model | Score | Response time (median) | Cost |
|---|---|---|---|
| Mistral Small 3 | 56.52% | 850 ms | $0.0016 |
| Llama 3.1 8B Instruct | 26.09% | 941 ms | $0.0019 |
The catalog was right about speed: 850 vs 941 milliseconds, a gap of 91 milliseconds, practically imperceptible. What no catalog predicted was the 30.43-point gap in answer quality. And Llama's four-times-larger context window did not help it at all here, because the items are a single sentence each.
This entire test, both models, 414 calls, cost $0.0035. Three and a half thousandths of a dollar to learn that two identically priced options are two times apart.
Our own answer key penalized correct answers
This is the part we least expected, and we nearly published the wrong numbers because of it.
The first run gave both models lower scores. Numbers that low made us suspicious, so we read through every answer scored as wrong. Some were genuine failures: the model left tp where it should have been tapi. But four items had a different problem, and the problem was on our end.
Those four items contained abbreviations that were not yet normalized in our own answer key:
"Jangan lupa bawa bku besok pagi." The word "bku" was never corrected to "buku".
"Jangan lupa bawa buku besok pagi." Completely correct, and scored WRONG.
The system sided with the broken key. Both models scored zero on all four items, and the model that normalized more correctly was penalized more heavily.
Four out of 69 items sounds small. What makes it serious is not the count but the direction: this error does not randomize; it penalizes the side doing its job correctly. The answer key has been fixed and all numbers in this article come from the post-fix run.
What found it was not checking items one by one, because all four had already passed human review. What found it was reading the answers scored as wrong, then asking why an answer that looked correct got a zero.
The number that matters more than the overall score
Fifteen of the 69 items are control items: the sentence is already standard and should be returned unchanged. A model that changes it is damaging writing that did not need fixing.
| Model | Correct sentences preserved |
|---|---|
| Mistral Small 3 | 82.2% |
| Llama 3.1 8B Instruct | 42.2% |
Llama damaged already-correct sentences on nearly six out of ten occasions. Mistral on two out of ten.
If this model were used as a writing assistant, this column is what matters, not the overall score. People forgive missed corrections, because they do not know what was missed. People do not forgive correct writing that gets ruined, because it is visible, and every time it is visible, trust in the tool erodes.
Also note the gap widens in this column: 30 points in the overall score, 40 points in control items.
For the first time, our set was large enough
There is a methodology note we are glad we can write this time.
In our previous measurements, most of our sets actually did not have enough items to prove the gap they were reporting themselves. This time it passed:
| Items needed for 80% statistical power | 37 |
| Items available | 69 |
The reason is not that this set got larger, but that the gap is large and both models err fairly often. That runs counter to intuition: the more often models are wrong, the fewer items you need to prove which one is better. In our other set, where both models are almost always correct, a gap that small demands 1,710 items.
So, which one
For normalizing Indonesian text, on the exact same budget and the same speed, Mistral Small 3 from Mistral AI gets it right twice as often as Llama 3.1 8B Instruct from Meta, and ruins already-correct sentences three times less often.
What must not be concluded from this: that one model is "better" in general. What was tested is one task type in one language. Llama has a context window four times larger, and on work that genuinely needs long context, this comparison does not apply at all.
What anyone can take away: price and speed rankings in vendor catalogs do not predict answer quality. To find out, the only way is to test it yourself on the work you will actually be doing. Ours cost three and a half thousandths of a dollar.
Data & provenance
Limitations
Two models, one set of 69 items, one language, one task type. Normalizing abbreviated writing is not a general measure of Indonesian language ability, and the results do not automatically apply to other tasks such as summarization or question answering.
This set is large enough to prove a gap of this size, not a small one. Two models that are close together still demand a much larger set.
Llama 3.1 8B Instruct's score moves 3.55 points between runs, three times that of Mistral Small 3. The average of three runs hides that instability.
Four corrected answer keys raised both models' scores. The numbers before and after the fix are not comparable, and this article uses the post-fix numbers.
A 91-millisecond gap in response time is too close to conclude anything about the latency rankings listed in vendor catalogs.
These items have never been published, but we have no way to prove the models have not seen the same sources.
Revision history
- Published
- Last updated
- never revised since publication
Terms in this article
- context window
- The limit on how much text a model can hold at once in a single exchange, counted in tokens. Often mistaken for a measure of intelligence when it is really a measure of capacity: a large window lets a model read long documents, but does not by itself make its answers more accurate on short ones.
- edit-based scoring
- Scoring an answer by comparing the EDITS it made to the original sentence, rather than matching the answer text character by character. What is compared is which words were removed and added. A correct answer that uses different quotation marks or spacing is no longer penalised, whereas exact text matching would have marked it wrong.
- error rate
- The percentage of wrong answers out of everything a model attempted on a test set. The inverse of score: a 91.7% score means an 8.3% error rate. Used instead of score when the point being made is how often a model gets things wrong rather than how often it succeeds, especially when comparing how far apart two models are.
- latency
- The time from sending a request to receiving the complete answer. It is not the same as the model's typing speed: this number also contains the wait when a provider is overloaded, so a handful of unlucky calls can pull the average far above the everyday experience.
- statistical power
- The chance that a test actually detects a difference that really exists. It depends on how large the difference is and how many items there are: a big gap can be shown with few items, a small one needs many. A test set that is too small can report two models looking different without being able to show the difference is not chance.
- temperature
- A dial that controls how willing a model is to pick a less likely word. Set to zero, it always takes the most likely option, so the same question tends to get the same answer. Raised, answers become more varied but harder to repeat. Benchmarks use zero so what gets measured is the model's ability rather than the luck of its sampling.
Written by the author for this article, not taken from a dictionary entry.