We test them ourselves, then publish the numbers together with the raw data and their limits. Including when the results are not what we hoped for.
Built on a dictionary of everyday speech and regional dialects that we curate ourselves.
We write the test items ourselves, and hold some back so they stay out of training data.
We tested two free NVIDIA models on 42 Indonesian slang questions. Nemotron Super, 120 billion parameters, answered 34 correctly; Nemotron 3.5 Lightning answered 26; both still trail Gemma at 31 billion parameters, priced at pocket change.
The two smallest models we have ever tested, LFM2.5 at 2.6 billion parameters and hy-mt2 at 1.8 billion, faced off on 60 Indonesian slang questions. The scores ended in a dead heat, but the two failed in opposite ways, and one of them tried calling Google mid-exam.
We tested 10 AI models on 135 Javanese words and predicted the results would split into two clusters with an empty gap between them. The last 3 models were picked precisely because they were the likeliest to break that prediction, and 2 of them landed squarely inside the gap we had called empty.
Three earlier articles ended with the same admission: every time we read one model's answers and fixed the key, that model gained the most. This time the protocol was designed to make that impossible, and across 135 Javanese words and 6 AI models, the protocol held. What we did not expect: once the instrument stopped moving, the AI models themselves turned out to be unsteady. At temperature 0, which should produce identical answers, 36 percent of answers changed between two consecutive calls.
Rankings on our Javanese benchmark have so far been set by a human-written answer key. To check whether that ranking is real, we added a second instrument that never sees the key: IndoBERT, a small 40 MB model that runs on our own machine. All four models came back in the exact same order. But 63 percent of the similarity it measured turned out to be sentence framing, not meaning, and that has to be measured first before a single number can be read.