AI benchmark·Empirical

Asked for a Single Letter, One AI Used 876 Times More Tokens

Imagine 7 people given a multiple-choice item that needs only one letter, but 3 of them write an essay first. That is how the 7 AIs we tested behaved, and the biggest gap reached 876 times.

This is a summary and another perspective on the research article.

One letter, 876 tokens

On 25 July 2026 we gave 7 AIs the same multiple-choice items from the Javanese speech levels benchmark. The answer requested was just one letter, whether A, B, C, or D.

Gemini 2.5 Flash
1.0
Gemini 3.1 Flash Lite
1.0
DeepSeek V3.2
1.9
Gemma 4 31B
1.9
Qwen 3.5 Flagship
726.0
Kimi K2.6
733.1
Qwen3 235B
875.8
Output tokens per single-letter answer

Of those 7 AIs, only Gemma 4 31B now has a model page on ibahasa.

There is no middle group. 4 AIs used fewer than 2 tokens, while 3 AIs used more than 700 tokens. None fell in between. For the exact same work, the most wasteful AI used 876 times more tokens than the most frugal.

The cause is not a mystery. Reasoning-type AI builds a chain of thought before answering, and that chain gets billed too, even when the requested answer is a single letter. The AI has no way to know the question was trivial.

Catalog prices do not predict the real bill

API prices are quoted per token, so it is natural to compare AIs by that number. The problem is that the number is only half the multiplication. The other half is how many tokens the AI actually uses, and that is written in no catalog anywhere.

In other words, two writing services charge the same rate per page. That rate does not tell you whether one of them will write 1 page or 700 pages for the same question.

An AI with a cheaper price per token can end up far more expensive, and you will not know until you run it yourself.

In our data, Qwen 3.5 Flagship and DeepSeek V3.2 sit in a similar price-per-token class. Qwen used an average of 726 tokens per answer, while DeepSeek used 1.9. If a budget is built from catalog prices alone, it can be off by hundreds of times.

We got it wrong once, then fixed it

The first version of our technical article put the gap at about 2,291 times, not 876. That figure was real, but it was pulled from 2 different runs. One of those runs had only 8 calls per AI, the smallest sample we have. On a set that was genuinely the same, that AI used only 726 tokens, not 2,291.

We corrected it as soon as it was caught. The mistake was not in the data but in us, picking the most striking number without checking whether it was comparable to the rest.

Token counts shift between test runs

Tracing that error led to a finding more useful than the original one. The same AI, tested again at a different time, can use very different token counts. We logged one AI using between 726 and 2,291 tokens across 5 test runs, so its highest figure was more than 3 times its lowest.

That means a single test run is not enough to estimate an AI's cost. Any single cost figure deserves suspicion, including the first version of our technical article and the 876 in this one, which also comes from a single run. Read 876 as an order of magnitude, not an exact number.

The pattern holds on newer models

This run is from July, and the models have since changed. The pattern held. On 60 slang items, DeepSeek V4.1 Flash used 21,239 output tokens, about 4 times the 4,981 of its predecessor, even though their scores differed by only 1 item (details here). On Javanese words, the top-scoring model turned out to be 187 times more expensive per correct answer than the model that trailed it by 11 words.

How we measured it, and all the raw data, are in the technical version of this article.

Behind this finding

The technical version has the raw numbers, the test setup, and everything that cannot be concluded from them.

Read the technical version