We test how well AI actually understands Indonesian and its regional languages, map the words that trip up native speakers, and publish the numbers along with the raw data and their limits, including when the results are not what we hoped for.
We put 4 AI models through the same 69 sentences under three different prompt wordings. The second wording raised one model from 32 to 59, and that wording contradicts our own answer key in 40 places.
The hypothesis was reasonable: let a contextual language model pick the best correction candidate based on sentence meaning. Tested A/B with a single switch and an answer key of 1,500 pairs, suggestion accuracy instead fell from 88.3 percent to 77.1 percent, while the model consumed 98.5 percent of analysis time.
We pointed our own spelling checker at writing people actually produced, not at test sentences. It flagged 3,740 things, and most were not errors. This is the record of eight rounds of fixing it.