How We Test: The Methodology Behind Every Number We Publish

Two different questions

When people ask "how good is AI at Indonesian", there are actually two different questions hiding inside that, and we measure them in two ways that cannot substitute for each other.

A model can be strong at one and ordinary at the other, and that is exactly what we want to show, not hide behind a single blended score.

The tools behind it

The numbers we publish do not come from one tool. They come from several that lean on each other.

Corpus

Before anything can be tested, there has to be material to test with. We collect real Indonesian and regional-language usage, not invented examples, as raw material for both the dictionary and the test data.

Dictionary and corrector

Our internal correction tool checks Indonesian spelling and grammar. It is judged not by how often it is right on examples we picked ourselves, but by how often it is wrong on real text it has never seen, because a correction tool tested only on easy examples will always look flawless.

One component uses a quantised (ONNX) IndoBERT model to rerank spelling suggestions. We measured this path as more accurate but also considerably slower, so it currently sits disabled in production until that balance makes more sense.

Test data (Benchmark)

This is what tests third-party AI models on closed questions, spanning standard spelling, normalising informal register, and regional speech levels. We write the questions ourselves from our verified corpus, and a portion is deliberately never published.

Data layer audit

Separate from the three above, there is a check on our own dictionary: whether entries, meanings, examples, and the relationships between words actually hold together, checked directly against what is running in production. This looks inward, not at an outside AI model.

AI Battle (Arena)

This is where readers do the judging. Two models answer the same question with no names shown, and you pick the better answer. The results build a model ranking for Indonesian, and going forward we plan for the arena's results to also feed research articles, not stand alone as just a leaderboard.

Rules we hold ourselves to

• Test data is written from our own verified corpus, not lifted from benchmarks already circulating online. A model cannot have memorised a question that did not exist before we wrote it.
• A portion of items is deliberately never published, so it stays usable for checking whether a model has seen the material before.
• Model names are hidden in AI Battle until you vote, and which side an answer appears on is randomised by the server, so what gets judged is the writing, not the brand.
• Every published figure links to its data file in a public repository, pinned to the version it was cited at.

Want to check the data file yourself, or cite the exact version we cited? It's all here, licensed CC BY 4.0.