A wrong prompt raised AI scores by 27 points

One task, one key, three ways of asking

Our test set id-N holds 69 Indonesian sentences written the way people write text messages, complete with abbreviations and dropped letters. The models have one job, namely returning each sentence to its standard form. The sentence Hrs dikerjain skrg has exactly one correct answer, and we wrote that answer before any model was called.

After running 4 models we looked at the answers scored wrong. Gemma 4 31B got 37 of 69 wrong, but 23 of those meant the same thing as the key and differed only in word choice. Gemma wrote Saya ingin tahu mengapa dia marah where the key says Aku ingin tahu kenapa dia marah. Not a misunderstanding, but a step further than we asked for.

We blamed our own prompt and wrote a second wording. So that the two could be compared, we moved the instruction out of the data, where it used to sit inside each item. That left the old wording intact and ran the new one over exactly the same 69 sentences and the same key.

Our key did not work the way we assumed

We built the second wording on one assumption, namely that the id-N key restores abbreviations but never replaces a word. The assumption was wrong, and we only found out after counting all 69 source-and-key pairs word by word.

Our key replaces words, 40 times, and it does so without a single deviation. A word that is already standard is never touched. aku survives 18 out of 18 times, kamu 6 out of 6. A word that is not standard is always replaced. gak 7 out of 7, then pengen, bgt and seneng 5 out of 5.

So the rule is a single one, and it is exactly what the old prompt said: make every word standard, and leave alone the ones already standard. A model that writes Saya for Aku really is wrong, because aku is already standard.

One score, two abilities

Once the rule was clear, the id-N score could be split into two abilities that can move in opposite directions. We read both of them off the key, not off a list of standard words of our own making.

A, the 40 substitutions the key makes. Does the model turn gak into tidak?

B, the 63 words the key keeps. Does the model leave aku as aku?

That 63 is not every word the key keeps, but the ones at least one model actually replaced. Of the 243 words the key keeps, the rest, such as hari and buku, tempt nobody, and counting them would only dilute the denominator until every model looked obedient.

The second wording raised the score by trading ability

The second wording reads do not replace words with synonyms and do not change the level of formality. Gemma's score jumped 27 points.

Wording 1Wording 2Wording 3

Gemma 4 31B

32 · 59 · 60

Qwen3-30B-A3B

40 · 43 · 46

Ling 3.0 Flash

39 · 43 · 41

Mistral Small 3

39 · 38 · 38

Split into A and B, that jump stops looking like an improvement.

Wording 1Wording 2Wording 3

Gemma 4 31B

87.5 · 77.5 · 87.5

Qwen3-30B-A3B

77.5 · 57.5 · 80.0

Ling 3.0 Flash

80.0 · 56.4 · 82.5

Mistral Small 3

72.5 · 60.0 · 72.5

Wording 1Wording 2Wording 3

Gemma 4 31B

54.0 · 98.4 · 95.2

Qwen3-30B-A3B

81.0 · 88.9 · 84.1

Ling 3.0 Flash

55.6 · 87.3 · 71.4

Mistral Small 3

81.0 · 93.7 · 82.5

The second wording brings B close to perfect on all four models, and damages A on all four as well, without a single exception. That wording buys obedience and pays for it with ability.

From there the score differences stop being a puzzle and turn into arithmetic. Gemma gains 27 because its debt on B was the largest, and it lost only 4 substitutions on A. Mistral loses 1 because its debt on B was the smallest, so there was little to buy, while its loss on A was the full one. Ling and Qwen sit between the two.

It is now clear where that score came from. The second wording forbids models to replace words, while our key replaces words 40 times.

The models that obeyed the second wording most closely were the models that stopped doing the most of the work we asked for.

27

The third wording, the right one

The third wording states the rule as it is, namely turn every non-standard word into its standard form, and do not replace a word that is already standard even if it sounds casual. We deliberately gave no example words, since most B cases are aku and kamu and naming them would hand over the answer.

The result has a different shape from the second wording, even though Gemma's score is similar. A returns to where it started on all four models, and on Gemma and Mistral it is identical to the first wording, namely 35 out of 40 and 29 out of 40. B stays far above where it began. Gemma reaches 60 out of 69, the highest of the three wordings.

Notice what did not change. The correct wording adds no ability to any model; rather, it stops getting in the way. On identical sentences and an identical key, our own wording had been hiding 28 points of Gemma's ability, from 32 to 60. The ranking inverts with it. Under the first wording Gemma is last of the 4 models, under the third it is first, and Mistral is the one that drops to last.

We nearly reported the second one as an improvement

The second wording passed every check we normally run. Same items, same key, same scorer, small cost, and a higher score on 3 of the 4 models. Had we stopped at the score, this article would be titled "a better prompt raised scores by 27 points", and every sentence in it would be true except the conclusion.

What saved it was not suspicion but the split into A and B, and that split came out of our own mistake. We only counted the key word by word after our first assumption about the rule turned out to be wrong. Had that assumption happened to be right, the instrument that separates the two would never have been built.

We also tested the strongest competing explanation, that all four models were merely guessing and the differences were noise. It does not hold. The direction of the change in A is the same on all four models, and 6 sentences went unanswered correctly by every model under all three wordings. Two of those expose an exception in our own key, namely dikerjain and ngerti are left in the correct answers although neither is standard, so both break the rule we had just stated. We are reporting that rather than tidying the key up out of sight.

What these figures do not say

This is not a model leaderboard. All four were picked for being cheap and fast, and Gemma rising the most does not make Gemma the best model for normalisation.

Nor do these figures say that a more detailed prompt is always better. The third wording wins because it matches the key, not because it is longer. The second wording is also more detailed than the first, and it does damage.

Only one thing here holds beyond this set, and it has nothing to do with Indonesian, namely a score that goes up is no evidence that the prompt got more correct. As long as a score adds up more than one ability, a single change of prompt can raise the number while lowering the work actually being measured. Checking for that is cheap. Split the score into the abilities underneath it, read the split off the answer key, and see whether they all move the same way. This whole experiment, 826 calls across 4 models, cost $0.017.