We test how well AI actually understands Indonesian and its regional languages, map the words that trip up native speakers, and publish the numbers along with the raw data and their limits, including when the results are not what we hoped for.
Our slang test offered free answers to anyone reading options. Yet, AI models grabbed at most 1.00 out of 4.1 items.
Seven models tested on one and the same set. An 876-fold gap in output tokens for a single letter answer, and the numbers themselves move between runs.
Six of our 40 Javanese test items were dropped. Not because models answered them wrongly, but because the textbook rule contradicted what speakers actually accept.