36% of an AI's answers changed even though the question was identical

Every time we fixed the answer key, one model gained

We test how well AI models know Javanese vocabulary. The question is a single sentence: what does this Javanese word mean in Indonesian?

The way we score the answers is just as simple. For each word we prepare a list of Indonesian equivalents we accept as correct, and we call that list the answer key. An answer is marked correct if it contains any equivalent on that list.

The trouble starts when a model answers with an equivalent that is perfectly valid but never occurred to whoever wrote the key. We then read that answer, add its equivalent to the key, and recompute every score. The key becomes more complete, and that genuinely is an improvement.

But the model whose answers we read collects the entire benefit of that improvement. Every other model, whose answers we never read, stays punished by a key that is just as incomplete as before.

In our measurement of 43 rare Javanese words, a single round of reading raised GPT-5.6 Luna by 14 points on its own. Two other models did not move by a single point, not because they were worse, but because the equivalents we added happened to be the ones Luna used.

An instrument that improves only for whoever happens to be inspected is not an instrument, it is a mirror.

We changed the order: run every model first, only then touch the key

Closing that gap required no new technology. We only had to change the order of the work.

WRONG   run model A → read its answers → fix the key → compute scores

        run model B → compute scores with a key already bent toward A



RIGHT   run EVERY model

        read all of their answers in a single round

        only then compute every score with the final key

That order is cheap because scoring runs over answers that are already stored. Recomputing 810 answers with a changed key requires no call to any AI model, and finishes in seconds. So as long as every model gets a reading, no model is favoured by the order.

We tested this way of working on 135 Javanese words from our own dictionary, with 6 AI models and 540 equivalents averaging 4.00 per lemma.

Three methods we combined, and why none of them is BLEU

Many people assume that scoring an AI's answer means comparing its sentence against a reference sentence, the way BLEU does for machine translation. Our approach is different, and each of its three parts has its own name in the literature.

The task is called definition modeling. A model is asked to state the meaning of a word, and its answer is scored. This task has been a shared task at SemEval, and its papers name one application directly: producing dictionary data for low-resource languages. Exactly our work.

The scoring is exact match with a set of aliases, standard practice in evaluating question-answering systems. An answer counts as correct if it contains one of the accepted answers. Researchers there have long acknowledged something we experienced ourselves: supplying a list of aliases loosens a rigid score, but it adds to the burden of collecting the data.

The key structure most resembles bilingual lexicon induction, which scores a single word's translation against a gold dictionary holding more than one valid translation. One source word, many equivalents, correct if any one of them lands.

So why not BLEU. BLEU scores one translated sentence against one reference sentence, while a single sentence can be translated many equally correct ways. That objection is what sank our first attempt. The question "what does this word mean" is free of it: the answer is right or wrong, and it cannot end in a tie.

What score would a model get if it just guessed?

That question has to be answered before a single figure is reported, and here is why.

Adding equivalents to a key raises scores. Adding equivalents also raises the odds of an answer landing by pure chance, with the model understanding nothing. Those two effects look exactly alike in a results table: the number goes up. So adding keys is never safe without one companion measurement.

We measure it this way: take a model's answer for one lemma, then test it against the keys of every other lemma. Pairs like that are guaranteed to be wrong. Anything that gets through is pure luck, and how much gets through is our zero point.

The result: 694 of 108,540 pairs, or 0.6 percent. We recomputed it every time the key was edited, and across six rounds that figure never rose.

The same check filtered out three kinds of broken key. The most embarrassing: a key whose content was the lemma itself, written twice. A model always repeats the word it was asked about, so that key let any answer through, including answers that were completely wrong.

Six rounds of fixing the key, and no model gained an advantage

Here is the evidence that changing the order of work actually did the job.

We ran six rounds of reading, one round per model. Each round improved the key, and each improvement applied to every model. This is what happened to the scores:

Compare that with the earlier measurement, where one model rose 14 points on its own while two others stayed put. This time every model rose, none fell, and none rose far beyond the others. The largest gain is 4 points, the smallest 1 point. And the model that gained least is the one with the highest score, which had the fewest errors left to fix in the first place.

A second check ran independently alongside. Throughout the fixing, we counted how often the key scorer and the semantic similarity scorer reached the same verdict. That second scorer is IndoBERT running on our own machine, and it never sees the answer key at all. The two agreed more and more often:

before review        0.879

after 3 rounds       0.896

after 6 rounds       0.913

Two scorers that share nothing agreed more often each time the key was fixed. That is a sign that what we improved really was the instrument, rather than simply loosening the scoring until everyone passed.

We ran one model three times, and its score came out different

Once our instrument stopped shifting, we called one model repeatedly to confirm its result was stable. The setting is temperature 0, which asks a model to always pick the most likely word. The questions were identical. The set was identical.

run 1   25 of 135

run 2   21 of 135

run 3   21 of 135

That model's score moved by 4 lemmas. We then compared its answers one by one between the last two runs:

identical answers :  86 of 135  (64%)

DIFFERENT answers :  49         (36%)

More than a third of the answers changed between two consecutive calls, at a setting that is supposed to guarantee the same output.

Most of those changes were cosmetic, the same wrong answer worded differently. But some crossed the line between right and wrong, and those are what moved the score.

temperature 0 never did promise what people think it promises. The setting governs how a model picks its next word; it does not guarantee that the provider runs an identical computation. Large models served alongside other users' requests do drift.

Uncertainty of that size lands directly on our own figures:

The gap between those two is 1 lemma. The between-attempt variance we just measured is 4 lemmas. Which means these two models cannot be told apart by this data, and calling either one better amounts to reporting noise as a finding.

We spent days closing gaps in our instrument. It turned out the larger source of uncertainty was not in the instrument at all, but inside the model being measured.

The measurements

All 810 calls cost $0.41, and 90 percent of that fell on one model alone.

Two things stand out, and neither can be guessed from the price.

Gemma 4 31B matches a model 3 times its price. It trails GPT-5.6 Luna by 1 lemma, at $0.0000133 per call against $0.0000418.

Claude Haiku 4.5 trails Gemma by 23 points while costing 14 times more. Price order and score order do not line up at all except at the two extremes.

What these numbers do not establish

This is not a general Javanese ranking of these six models. What we measured is recognition of single-word meanings. Translating sentences, choosing the right speech level, or writing in Javanese are different abilities, untouched here.

A gap of 1 to 4 lemmas is not meaningful. The between-attempt variance we measured is 4 lemmas. The top four models sit within a range of 14 lemmas, and most of the neighbour-to-neighbour gaps inside it are smaller than that noise.

A high score does not prove command of Javanese. Our dictionary pages are publicly indexed, and these 135 lemmas are not rare words. Any model may well have read them.

Our answer key is not complete. The semantic scorer measures the remaining gap: 14 percent of the answers we scored wrong turn out to sit very close to our own dictionary definitions, and merely fail to name the word that happens to be listed in the key. Adding equivalents narrows that gap; it never closes it.

The between-attempt variance is not necessarily the same for every model. We measured it on a single model, the weakest one. Weak models sit near decision boundaries, so their answers flip more easily.

What holds beyond Javanese

Reading order is part of the methodology, not a matter of tidiness. As long as fixes to an instrument are guided by one participant's output, that participant gains an advantage that will never appear in the results table. The remedy is not to stop fixing, but to fix after every participant has been tested.

Measure what score is achievable while understanding nothing, and measure it again every time the scorer changes. Without that figure, a high score cannot be told apart from a scorer anything can slip past.

Do not assume temperature 0 guarantees a repeatable answer. If the gap between two systems is smaller than a between-attempt variance nobody has measured, there is no ranking between them yet. Measuring it is cheap: run it twice, then compare.