Why this matters
Javanese has speech levels that change the word itself, not just its tone. The word for "eat" changes form depending on who you are speaking to: mangan to a peer, nedha to someone you respect, dhahar to someone of much higher status. These are not three interchangeable synonyms. One is correct for a given situation, and the other two are wrong for that same situation.
Picking the wrong level is not a grammar mistake, it is a manners mistake, and speakers notice immediately. That is what makes this ability different from simply memorizing vocabulary: an AI model has to know not just what a word means, but also who it is appropriate to say it to. As more people use AI to write or translate Javanese, measuring this ability matters more, rather than assuming it.
We tested two models from Anthropic on this ability directly: Claude Haiku 4.5, the cheapest tier, and Claude Sonnet 4.6, the expensive tier from the same company. Claude Haiku got 10 of 36 questions wrong. Claude Sonnet got only 3 wrong.
What we found
Both models worked through the exact same question set: 36 multiple-choice questions on Javanese speech levels, choice order shuffled per question, three repeated runs. All three runs gave identical answers for both models, so the gap we found is not a one-time fluke.
Claude Haiku 4.5 answered 72.22 percent of questions correctly, at a cost of $0.026 for its entire test run. Claude Sonnet 4.6 answered 91.67 percent correctly, at a cost of $0.079, three times more expensive than Claude Haiku for the same number of questions. The score gap is 19.45 percentage points, and testing both models together, 264 calls in total, cost only $0.11.
This gap clears the usual statistical significance threshold, but it is thin: a single question flipping its answer would already be enough to shift this conclusion.
The biggest temptation with numbers like these is to conclude that the more expensive model always understands regional languages better. The data does not support a claim that broad. In our previous testing on this same skill, a much cheaper free model actually outperformed Claude Sonnet. What this test shows is only that these two particular models, from the same company, are far apart on this one ability.
The limits
There is good news on the other side. Regional languages are often assumed to be equally hard for any AI to crack, and this test partly disproves that: Claude Sonnet answered 33 of 36 questions correctly. Javanese speech levels are not an impossible wall for AI, as long as the model's underlying ability is good enough. Even the cheapest model, Claude Haiku, still answered 72.22 percent of questions correctly, far from just guessing.
What remains unanswered: whether the same pattern holds for other AI companies, for other regional languages, or for questions in a format other than multiple choice. Eight fill-in-the-blank questions on the same set were also run, but have not yet been scored because they are still waiting for a Javanese speaker to check them, so the numbers in this article do not yet cover the full ability we actually want to measure.
Two more things worth knowing before using these numbers. Our own statistical power calculation says a gap this size would ideally need around 57 questions to measure with confidence. We only have 36, so treat this as early evidence, not a final number. Both models were also called directly through Anthropic, while most of our other tests go through one intermediary, so the Claude Haiku versus Claude Sonnet comparison here is clean, but its comparison to models from other companies in this rubric is not always apples to apples.
The full method, repeated runs, and raw data are in the technical version of this article.
Behind this finding
The technical version has the raw numbers, the test setup, and everything that cannot be concluded from them.