We test how well AI actually understands Indonesian and its regional languages, map the words that trip up native speakers, and publish the numbers along with the raw data and their limits, including when the results are not what we hoped for.
Our slang test offered free answers to anyone reading options. Yet, AI models grabbed at most 1.00 out of 4.1 items.
Mistral Small 3 and Llama 3.1 8B Instruct are sold at the exact same price and are equally fast. On 69 items testing normalization of abbreviated writing to standard Indonesian, one scored 56.52%, the other 26.09%.
The hypothesis was reasonable: let a contextual language model pick the best correction candidate based on sentence meaning. Tested A/B with a single switch and an answer key of 1,500 pairs, suggestion accuracy instead fell from 88.3 percent to 77.1 percent, while the model consumed 98.5 percent of analysis time.
Two Anthropic models were tested on 36 Javanese speech level questions. Claude Sonnet 4.6 missed 3, Claude Haiku 4.5 missed 10.
Seven models tested on one and the same set. An 876-fold gap in output tokens for a single letter answer, and the numbers themselves move between runs.
A 5.8% pass rate. That number is the real price of verification, and the reason "just let AI handle it" is not an answer.
We pointed our own spelling checker at writing people actually produced, not at test sentences. It flagged 3,740 things, and most were not errors. This is the record of eight rounds of fixing it.
Six of our 40 Javanese test items were dropped. Not because models answered them wrongly, but because the textbook rule contradicted what speakers actually accept.