We pit IndoBERT against our own human answer key on the Javanese benchmark

The objection a better key cannot answer

We have now tried twice to measure how well AI models understand Javanese. The first attempt scored translations against a single dictionary gloss, and failed. The second replaced it with a question whose answer is binary, what does this word mean, then tested 4 AI models on 43 rare Javanese words. The scoring was simple: an answer counted as correct if it contained one of the Indonesian equivalents listed in our own answer key.

We wrote that key ourselves. That is where the problem lives.

If a model answers with an equivalent the key's author never thought of, it gets punished. We have already patched the key mid-measurement twice, and both times the score jumped without a single call being repeated. So the following objection is fair, and has to be answered:

Your ranking is not measuring the model. It is measuring how well the model's answer happens to match the words you wrote in the key.

That objection cannot be answered by writing a tidier key, because a tidier key is still a key. It can only be answered by a second instrument that works a different way and never sees the key at all.

IndoBERT, and why we picked a model this small

We use IndoBERT in ONNX form, a derivative of LazarusNLP/congen-indobert-lite-base. It is 40 MB, and the model has been in our code for a while already, used by our editor to rank synonym candidates.

IndoBERT's job here is exactly one thing: turning a sentence into a string of numbers. Two sentences with close meanings produce two strings of numbers that are close together, and that closeness can be computed. So we can ask "how close is this model's answer to our dictionary definition" without naming a single key word.

4 reasons we picked this kind of model, rather than asking another AI to serve as judge.

1. IndoBERT never changes its mind. The same input always produces the same numbers. We checked by running it twice in a row: the output files matched byte for byte. To check an instrument that has already proven itself capable of moving, a checker that holds still is worth far more than one that is slightly smarter.

2. IndoBERT is the right class of model for this job. It reads and summarizes, it does not compose. For comparing two pieces of text, that is the correct class of tool. Using a generative model for this task would be overkill, and harder to audit.

3. IndoBERT is trained for Indonesian. The definitions being compared are written in Indonesian, some of them containing Javanese terms and dictionary-register vocabulary.

4. IndoBERT is free and runs on our own machine. No quota, no queue, no per-call cost. It also works on answers that are already saved, so all four models got a new column without a single one being called again. The entire computation finished in seconds.

One similarity number means nothing without its floor

Before a single result can be reported, one thing has to be measured first.

Semantic closeness comes out as a number between 0 and 1. Say an answer scores 0.52. High or low? Nobody knows, until we know what that number looks like for pairs guaranteed to be unrelated.

So we measured it. Each lemma's answer was crossed against the definitions of the other 42 lemmas, giving 7,224 pairs guaranteed to be wrong:

Read the first row again. Pairs that are completely unrelated still score 0.389. The cause is the sentence frame both sides use. The model answers "The word X in Javanese means ...", the dictionary writes "An interjection in Javanese that ...", and the machine reads that shared shape as semantic similarity.

Strip the lemma and 49 framing words from both sides, and the number falls to 0.143. That means 63 percent of the similarity originally measured was not meaning.

We suspected the framing was concentrated in the opening clause, so we tested a gentler cleaning pass: strip only the opening phrase by pattern, leave the rest of the sentence intact. That guess missed. The floor barely moved, from 0.389 to 0.383. The framing is spread across the dictionary's phrasing, not stacked at the front.

One more thing surprised us: the instrument's separating power barely changed under any of the cleaning. Cleaning the text does not make the instrument smarter. It makes the number readable by a person.

Before the two were pitted against each other, the key had to be cleaned up

Comparing two instruments only makes sense if both are clean. In the earlier article, DeepSeek V4 Flash was the one model whose answers had never been read by a human. This time a speaker read them, answer by answer.

That reading found something we were not looking for.

For the lemma gremet, meaning to move very slowly, our key list included the entry gremet gremet. That is the lemma itself, repeated, because the reduplicated form is mentioned in our own definition. But a model always repeats the word it is asked about when it answers. So that key let any answer through:

Mistral Small 3     "Gremet berarti makanan yang enak dan lezat." (means: delicious, tasty food)  → scored CORRECT

DeepSeek V4 Flash   "berbutir-butir halus atau terasa kasar" (means: grainy or coarse-textured)     → scored CORRECT

Both are completely wrong, both got through, and both matched only through that one entry.

We have rejected a similarly flawed key before: the word jawa was once proposed as an equivalent, then rejected after it turned out to appear in 37 percent of all answers. gremet gremet passed the same check precisely because it never appeared in any other lemma. Our leak test compares across items, and this key only needed to let through one.

Here is what changed, and none of it changed because a model answered differently:

Of Mistral's two answers that were originally scored correct, one got through by way of the broken key. That model is actually correct on one lemma out of 43.

Two things happened here for the first time. A patch lowered a score, when the three earlier rounds had always raised one. And reviewing one model uncovered a flaw that had inflated a different model: DeepSeek is what got read, Mistral is what lost points.

The key scorer's chance floor dropped too, from 1.3 to 0.7 percent. Removing overly broad keys mattered more than adding new narrow ones. For the first time, a review round made the instrument stricter.

One proposal we turned down

Reading 43 answers raised questions that did not always have a clean answer.

Pimen is a non-standard form of the question word "how" in the Banyumasan dialect. DeepSeek answered "to ask, or asking". That is the word's grammatical function, not its meaning. The initial proposal was to accept bertanya as a valid equivalent, on the grounds that the answer was not entirely wrong, just imprecise.

That proposal was withdrawn, for the sake of consistency. In the same review session, the key bunyi was in fact dropped from the lemma klotekan for the exact same reason: it names the category of meaning, not the meaning itself. Dropping one while adding the other would make the criterion depend on which lemma happened to be under review.

There is a line that has to be held in a benchmark with binary answers. What is measured is whether the equivalent is named, not whether the answer is still in the vicinity. The moment "close enough" counts as correct, what is being measured changes, without anyone announcing it.

Gondes has two senses, and our dictionary records only one

The same sweep found 5 lemmas keyed on slang. Every Javanese slang word is slang, so that key passes an answer that says nothing at all about meaning. Every one of them was dropped.

Dropping them exposed something in the lemma Gondes. Our dictionary records it as a familiar form of address from Semarang, equivalent to "bro". But it comes from "gondrong ndeso", a term for a long-haired young man of rustic style, and that sense is still alive.

Gemini 3.5 Flash  "short for gondrong ndeso, a long-haired young man"   → WRONG

GPT-5.6 Luna      "a slang term, faintly derogatory, for a gondrong lad" → CORRECT

Gemini gave the most precise answer of the four, down to the etymology, and was scored wrong. Luna said the same thing less carefully and passed, purely because it happened to use the word "slang". The slang key had been covering the hole, not causing it.

The hole: our dictionary records one sense for a word that has two. The equivalents gondrong and ndeso were added, and both models now pass on the substance of their answers. That is the one figure that went up in the correction table.

This is the single-reference problem, the same class that sank our earlier translation design. There, one sentence had many valid translations; here, one lemma has more than one valid sense. And what surfaced is not only a benchmark flaw: our own dictionary entry is incomplete.

Two instruments, one order

Once the key was clean, the two instruments were pitted against each other.

The order came out exactly the same as the key's order.

The two right-hand columns come from IndoBERT and never touch the key. IndoBERT turns every answer and every definition into a string of 768 numbers, and their closeness is measured by cosine similarity, the angle between the two strings: the narrower it is, the closer the meaning.

Semantic percentile reads that closeness against the chance floor from earlier. A value of 91.6 percent means Gemini's answers are on average closer to their definitions than 91.6 percent of the 7,224 pairs guaranteed to be unrelated. Mistral's answers reach only 36.7 percent, below most of the pairs guaranteed to be wrong.

Semantic rank does not even use the chance floor. For each lemma the four answers are ordered by their closeness to the same definition; the top one gets 1, the bottom one gets 4, then averaged across 43 lemmas.

Across all 172 answers, the two instruments agree at a rate of 0.937. In other words: pick one answer the key calls correct and one it calls wrong at random, and 94 times out of 100 the correct one scores higher semantically.

This is not proof the ranking is correct. It only closes off one alternative explanation, the one people raise most often.

What this instrument cannot do

Similarity is not correctness. Bladak is a fritter from the Semarang region, and all four models made up different answers, not one of them correct. The semantic score still handed out a ranking, and the top one scored the 71.9th percentile for the answer "open, or clearly visible", which is pure invention. A high percentile means it sounds like the right definition, not that it is right.

As a filter, its usefulness is middling. We tested it back against the earlier review round. Of the 15 lemmas scored wrong at the time, 8 turned out to be instrument flaws. Reading the top 5 by semantic order finds 4 of the 8; a random order would be expected to find 2.7. Better, but the gap is one or two lemmas. It sets the reading order. It does not replace the reading.

Its strongest signal is exactly the one we cannot use here. Comparing an answer to the key list separates human verdicts better than comparing it to the definition, 0.968 against 0.930. But that stronger signal uses the key, so it cannot check whether our key is biased. The weaker signal is the useful one, precisely because it stands on its own.

That is why the semantic score never delivers a verdict. Right and wrong are still decided by the key, which anyone can check by reading 43 lines. The embedding score cannot be checked that way. The moment it enters the path to truth, our benchmark's result depends on a model, and that is exactly the property we demand an instrument not have.

What holds beyond Javanese

Three things, and none of them specific to rare Javanese words.

Check for answer keys that contain their own question. gremet gremet got through because our leak test compares across items, while that key only needed to let one item through. A key like that will never be caught by a test that hunts for leaks across items.

Measure the chance floor every time the key changes, in both directions. A key that widens raises the score and raises the chance of a hit at the same time, and both look identical in a results table. This time the key narrowed and the floor dropped, and even that was only known after it was measured.

A similarity number without its floor cannot be read. Two-thirds of the similarity we measured turned out to be sentence framing. Anyone who uses a similarity score like this as a judge, without first measuring it on pairs guaranteed to be wrong, is counting style as truth.