Seven AI models tested on a single letter answer with an 876-fold token gap

We gave seven models a multiple choice test on one and the same item set. The answer asked for was a single letter. A, B, C, or D.

Then we counted the output tokens each spent producing it. The leanest used 1.0 token. The heaviest used 875.8. A factor of 876, for exactly the same work.

One letter, 875 tokens

Every figure below comes from a single run, 2026-07-25-jv-A-procurement, with 32 calls per model. Keeping them inside one run matters, and the third section of this article explains why.

There is no middle group. Four models sit below two tokens, three sit above seven hundred, and nothing sits between them. This is not a spectrum, it is two populations that happen to have been given the same task.

The mechanism is no mystery. Reasoning models produce a chain of reasoning before answering, and that chain is billed. What is surprising is not the mechanism but that it still runs when the answer requested is a single letter. The model has no way of knowing the question is trivial.

How we measured it

Each run is exported to a file holding per-call results: model, item, score, input tokens, output tokens, and format failures. The figures in the table are mean output tokens per call, computed across 32 calls per model.

Why output tokens rather than currency: prices change at any time and differ between providers, while token count is a property of the model's own behaviour. It will still be true a year from now even after the price list has changed three times.

A choice that can be argued with: we compared on one format only, multiple choice. That format is the extreme case for reasoning models, because it maximises the ratio between reasoning and answer. On a task like writing a paragraph, the gap is almost certainly far smaller. These figures show an upper bound, not typical daily usage.

The first version of this article was wrong, and here is how

The table above now comes from a single run. The first version did not.

We reported 2,290.8 tokens for qwen-3.5-flagship and claimed a factor of 2,290. That number is real, but it came from a different run than the other six figures in the same table. That run, 2026-07-25-jv-A-produksi, had only eight calls per model, the smallest sample of any run we hold.

On the genuinely same set, that model used 726.0 tokens, not 2,290.8.

The phrase "on the same set" in the first version was therefore untrue, and the headline figure overstated the gap by roughly a factor of three. The fault was not in the data. It was in us taking the most striking number from a pile of runs without checking whether it was comparable to the rest.

The numbers themselves move between runs

Chasing that mistake produced a finding more useful than the original one.

The same model, on different runs, spends very different amounts:

qwen-3.5-flagship ranges from 726 to 2,291, a spread wider than its own lowest value. That means a single measurement of a reasoning model is not enough to estimate its cost, and any single figure you read anywhere, including in the first version of this article, deserves suspicion.

Part of that spread has an ordinary explanation: different runs use different item sets and different prompt lengths. But we did not measure how much of it those causes account for, so the range should be read as total spread, not as the model's pure variance.

Why catalogue prices mislead

API prices are quoted per token, so it is natural to compare models by looking at that number. The problem is that the number is only half of the multiplication.

The other half is how many tokens the model actually spends, and that appears in no catalogue anywhere. No provider labels a model "tends to spend 700 tokens answering a trivial question".

A model with a lower per-token price can end up far more expensive, and you will not know until you run it yourself.

The example is in the first table. qwen-3.5-flagship and deepseek-v3-2 sit in price classes that are not far apart per token. In practice, on the same set, one spent 726.0 tokens and the other 1.9. Budget from the catalogue and you will be wrong by a factor of hundreds.

This is also why we now pick models from real usage logs rather than from catalogues. A catalogue tells you the price; a log tells you the bill.

And with a between-run spread that wide, one run is not yet a log worth trusting.