Why This Matters
AI leaderboards get read as lists of facts. This model 60, that one 38, so the first is smarter. What rarely gets read along with them: the number is not a property of the model, but the result of a meeting between the model and a question a human wrote.
We built a leaderboard like that for a simple task, turning 69 text-message style sentences into standard Indonesian. Then we inverted it ourselves, without touching a single question and without changing a single word of the answer key.
What We Found
We put the same 69 sentences to 4 models under three different wordings of the instruction. The ranking flipped at both ends.
60
46
41
38
Under the first wording, Gemma 4 31B from Google is last of the 4 models. Under the third it is first, and Mistral Small 3 from Mistral AI is the one that drops to last. Same questions, same key, same scorer. All that changed was an instruction sentence we had written ourselves.
The cause is not prompt magic but our own mistake. The first wording told models to make the sentence standard, and Gemma obeyed too far: it wrote Saya where the key writes Aku, although aku is already standard. Of Gemma's 37 answers scored wrong, 23 meant the same thing as the key and differed only in word choice.
Our own first fix nearly made things worse. We wrote a second wording that forbade models to replace words, and Gemma's score leapt to 59. We came close to announcing it as a better question, before realising the wording broke our own answer key in 40 places. The score rose for the wrong reason.
We then counted our answer key again, word by word. It restores abbreviations 128 times, and it replaces words 40 times: gak becomes tidak, pengen becomes ingin. The rule turns out to be a single one, make every word standard and leave alone the ones already standard. The third wording states that rule plainly, and Gemma immediately reaches 60 out of 69.
So Gemma was never the one unable to do the task. Our question was the unclear one. Both produce exactly the same low number, and the number itself never says which. A leaderboard reader sees only the number.
Mistral, meanwhile, held at 38 to 39 across all three wordings. For that one model our question was never the obstacle, and that is precisely what pushed it down the ranking once the other models were freed from theirs.
The Limits
What inverted here is our leaderboard, on one task, in one language. This is not a claim that Gemma is the best model or Mistral the worst. All four were picked for being cheap and fast so that three full wordings stayed affordable, and there is no large model among them.
Do not read the middle of the ranking too closely. Under the first wording Mistral and Ling both score 39, and under the third the gap from Ling to Mistral is only 3 points. Each sentence was asked once, with no repeats, so a difference that small could be noise. The only movement large enough to read as a finding is the 28 points on Gemma.
Six sentences went unanswered correctly by every model, under all three wordings. Two of them expose an exception in our own key, and we chose to report that rather than quietly tidy the key up.
Only one thing here holds beyond this set, and it has nothing to do with Indonesian: a ranking that the person writing the questions can invert is not a ranking of ability. Before trusting an AI leaderboard, ask who wrote its questions and whether that wording was ever tested. This whole test, 826 calls across 4 models, cost $0.017.
Behind this finding
The technical version has the raw numbers, the test setup, and everything that cannot be concluded from them.