We test how well AI actually understands Indonesian and its regional languages, map the words that trip up native speakers, and publish the numbers along with the raw data and their limits, including when the results are not what we hoped for.
GPT-6 Astra scored 42 of 42 on the public half of our Indonesian slang benchmark, the joint best result in the panel. Counting the 18 withheld items too, it finished 59 of 60, 1 behind OpenAI's own cheaper GPT-5.6 Luna. 1 item apart on 60 is not a statistically meaningful gap, and that is the more interesting finding: the public half no longer separates the strongest models at all.
GPT-6 Astra, OpenAI's newest and most expensive model, answered 33 of 43 rare Javanese words correctly, behind Google's Gemini 3.5 Flash (40/43) and OpenAI's own cheaper GPT-5.6 Luna (35/43), at 56 times Luna's cost. On a separate Javanese speech-level test, Astra tied Gemini 3.5 Flash for the best score in the panel.
We tested the stealth model ox-alpha while it was free, then tested the official model Z.ai declared as its identity, GLM-5.3-Flash, on the same question sets. Scores rose from 49 to 54 of 60 on Indonesian slang and from 27 to 32 of 38 on Javanese speech levels. The rest stayed similar: answer match of 88% and 79%.
We put 4 AI models through the same 69 sentences under three different prompt wordings. The second wording raised one model from 32 to 59, and that wording contradicts our own answer key in 40 places.
We tested two free NVIDIA models on 42 Indonesian slang questions. Nemotron Super, 120 billion parameters, answered 34 correctly; Nemotron 3.5 Lightning answered 26; both still trail Gemma at 31 billion parameters, priced at pocket change.
The two smallest models we have ever tested, LFM2.5 at 2.6 billion parameters and hy-mt2 at 1.8 billion, faced off on 60 Indonesian slang questions. The scores ended in a dead heat, but the two failed in opposite ways, and one of them tried calling Google mid-exam.
We tested ox-alpha and GLM-5.3 on Javanese speech levels and Indonesian slang in a trial lane separate from the official leaderboard. GLM-5.3 led on all four measurements, and a 71% answer match shows the two are not the same model.
We tested 10 AI models on 135 Javanese words and predicted the results would split into two clusters with an empty gap between them. The last 3 models were picked precisely because they were the likeliest to break that prediction, and 2 of them landed squarely inside the gap we had called empty.
Three earlier articles ended with the same admission: every time we read one model's answers and fixed the key, that model gained the most. This time the protocol was designed to make that impossible, and across 135 Javanese words and 6 AI models, the protocol held. What we did not expect: once the instrument stopped moving, the AI models themselves turned out to be unsteady. At temperature 0, which should produce identical answers, 36 percent of answers changed between two consecutive calls.
Rankings on our Javanese benchmark have so far been set by a human-written answer key. To check whether that ranking is real, we added a second instrument that never sees the key: IndoBERT, a small 40 MB model that runs on our own machine. All four models came back in the exact same order. But 63 percent of the similarity it measured turned out to be sentence framing, not meaning, and that has to be measured first before a single number can be read.
Four AI models were asked the meaning of 43 Javanese words absent from NusaX, NusaWrites and Javanese Wiktionary. Mistral Small 3 got 5 percent right, DeepSeek V4 Flash 53 percent, GPT-5.6 Luna 81 percent, Gemini 3.5 Flash 91 percent. The instrument itself was patched twice mid-measurement, and each patch raised scores without a single call being repeated.
Example sentences from our Javanese dictionary became the test items, and their Indonesian glosses became the answer key. Two AI models were measured. In a blind adjudication of 20 pairs, our glosses won 4, the model won 5, and the remaining 11 were tied. An answer key that is no better than what it grades measures agreement on word choice, not ability.