We test how well AI actually understands Indonesian and its regional languages, map the words that trip up native speakers, and publish the numbers along with the raw data and their limits, including when the results are not what we hoped for.
The two smallest models we have ever tested, LFM2.5 at 2.6 billion parameters and hy-mt2 at 1.8 billion, faced off on 60 Indonesian slang questions. The scores ended in a dead heat, but the two failed in opposite ways, and one of them tried calling Google mid-exam.
We tested ox-alpha and GLM-5.3 on Javanese speech levels and Indonesian slang in a trial lane separate from the official leaderboard. GLM-5.3 led on all four measurements, and a 71% answer match shows the two are not the same model.
We tested 10 AI models on 135 Javanese words and predicted the results would split into two clusters with an empty gap between them. The last 3 models were picked precisely because they were the likeliest to break that prediction, and 2 of them landed squarely inside the gap we had called empty.
Three earlier articles ended with the same admission: every time we read one model's answers and fixed the key, that model gained the most. This time the protocol was designed to make that impossible, and across 135 Javanese words and 6 AI models, the protocol held. What we did not expect: once the instrument stopped moving, the AI models themselves turned out to be unsteady. At temperature 0, which should produce identical answers, 36 percent of answers changed between two consecutive calls.
Rankings on our Javanese benchmark have so far been set by a human-written answer key. To check whether that ranking is real, we added a second instrument that never sees the key: IndoBERT, a small 40 MB model that runs on our own machine. All four models came back in the exact same order. But 63 percent of the similarity it measured turned out to be sentence framing, not meaning, and that has to be measured first before a single number can be read.
Four AI models were asked the meaning of 43 Javanese words absent from NusaX, NusaWrites and Javanese Wiktionary. Mistral Small 3 got 5 percent right, DeepSeek V4 Flash 53 percent, GPT-5.6 Luna 81 percent, Gemini 3.5 Flash 91 percent. The instrument itself was patched twice mid-measurement, and each patch raised scores without a single call being repeated.
Example sentences from our Javanese dictionary became the test items, and their Indonesian glosses became the answer key. Two AI models were measured. In a blind adjudication of 20 pairs, our glosses won 4, the model won 5, and the remaining 11 were tied. An answer key that is no better than what it grades measures agreement on word choice, not ability.
Our slang test offered free answers to anyone reading options. Yet, AI models grabbed at most 1.00 out of 4.1 items.
Mistral Small 3 and Llama 3.1 8B Instruct are sold at the exact same price and are equally fast. On 69 items testing normalization of abbreviated writing to standard Indonesian, one scored 56.52%, the other 26.09%.
The hypothesis was reasonable: let a contextual language model pick the best correction candidate based on sentence meaning. Tested A/B with a single switch and an answer key of 1,500 pairs, suggestion accuracy instead fell from 88.3 percent to 77.1 percent, while the model consumed 98.5 percent of analysis time.
Two Anthropic models were tested on 36 Javanese speech level questions. Claude Sonnet 4.6 missed 3, Claude Haiku 4.5 missed 10.
Seven models tested on one and the same set. An 876-fold gap in output tokens for a single letter answer, and the numbers themselves move between runs.