Claude Haiku misses 10 of 36 Javanese speech level questions, Claude Sonnet only 3

Two models, one company, one test

Javanese has speech levels that change the word itself, not just its tone. "Eat" becomes mangan to peers, nedha to respected individuals, and dhahar to those of much higher status. Choosing the wrong level is not a grammatical error, but a breach of etiquette, and native speakers catch it immediately.

That is what we tested: not whether a model knows the meaning of a word, but whether it knows to whom that word is properly addressed. Two models from Anthropic, one budget tier and one expensive tier, completed the exact same set of questions: 36 multiple-choice questions on Javanese speech levels, with option orders randomized per question, three repetitions, and zero temperature.

The difference is 19.45 percentage points. The budget model was wrong more than three times as often as the expensive model from the same family, and testing both models entirely, 264 calls, consumed eleven cents.

This difference clears the usual statistical testing threshold (two-sided p around 0.03), but it is thin: a single question shifting sides is enough to tilt the conclusion.

We almost reported the results in reverse

Before reaching the scores, there was another figure we almost made our main finding, and fortunately we checked first.

Read as is, the mean response time for Claude Haiku 4.5 was 2,444 ms, while Claude Sonnet 4.6 was only 1,609 ms. The budget model appeared 52% slower than the expensive model from the same family. Interesting, counterintuitive, and wrong.

The cause was not model speed, but the network. Our timing figure measured the wall time of a single call, including wait delays when the provider was busy and requests had to be retried. Anthropic happened to be overloaded precisely during Haiku's turn: seven of its calls failed completely and had to be retried, and its slowest answer reached 23,897 ms. A tail that long drags the mean, not a slow model.

This is not a one-time occurrence. Measured across all 5,705 rows of our measurements, the mean exceeded the median by between 4% and 152% depending on the model. The lesson: on data with a long tail caused by network interference, the mean is a skilled liar.

What must not be concluded from these numbers

The biggest temptation from the table above is to conclude that "expensive models are better for regional languages." The data does not support a claim that broad. What is shown is only that two specific models, from the same company, are widely separated on this single dimension.

In our previous measurements on the same dimension, a budget-tier open model scored 96.88%, and a highest-tier model scored 100%, both above Claude Sonnet 4.6 in this test. The number of questions differed (32 versus 36) so they are not directly comparable, but it is enough to forbid the conclusion that price order is capability order.

The good news, on the other hand: regional languages are often assumed to be an area where all AI models are equally bad, and this test refutes that. Claude Sonnet 4.6 answered 33 of 36 Javanese speech level questions correctly. Javanese speech levels turn out not to be an impossible wall for AI to scale, provided its foundational capacity is genuinely adequate.