Mercury 2.5 Is Fast, but Got Only 13 of 43 Rare Javanese Words Right

Two new models, the same four tasks

Inception released Mercury 2.5 on 8 September 2026. Mercury is a diffusion model. Models of this kind do not write an answer word by word from left to right. They draft a rough version and then refine many parts of it at once. Inception claims a speed of 1,107 tokens per second and intelligence on par with several models, including GPT-5.6 Luna (Low) and Claude Haiku 4.5. Two days later DeepSeek released DeepSeek V4.1 Flash, the direct successor to DeepSeek V4 Flash, with MIT-licensed open weights.

In other words, an ordinary model types its answer from the first letter to the last, while a diffusion model writes the whole paragraph at once as a blur and sharpens it over and over.

We ran both through OpenRouter on 15 September 2026 on four tasks from three benchmarks: 60 multiple-choice items on Indonesian slang, 38 multiple-choice and 8 open-answer items on Javanese speech levels, and 43 rare Javanese words. Each task was run once. Every score in this article counts the whole set, the public slice plus the held-out slice we never publish. Open answers were graded by a native speaker the following day.

Before a single call went out, we locked 6 predictions. Of those, 4 hit and 2 missed.

The slang token count in the last prediction includes 2 items retried with a higher cap, so part of that miss comes from the retries.

Mercury 2.5: about 1 second on every task

Mercury 2.5's median latency stayed around 1 second on all four tasks: 1,041 ms on slang, 979 ms on multiple-choice speech levels, 992 ms on open answers, and 1,056 ms on rare Javanese words. Other models in the panel swing far more from one task to the next.

Indonesian slangSpeech levels multiple choiceRare Javanese words

Mercury 2.5

1.0 · 1.0 · 1.1

Gemma 4 31B

0.6 · 0.8 · 1.9

GPT-5.6 Luna

1.1 · 1.0 · 3.7

GPT-6 Astra

2.8 · 1.9 · 4.3

DeepSeek V4.1 Flash

2.7 · 4.3 · 8.3

Mercury is not always the fastest. Gemma 4 31B is quicker on slang and on speech levels. Mercury's edge is its stability. Rare Javanese words slowed GPT-5.6 Luna down by almost 4 times compared with speech-level items, from 988 ms to 3,708 ms, while Mercury barely moved.

This pattern is the opposite of GPT-6 Astra, which we covered in an earlier article. Astra is frugal with tokens but was never the fastest. Mercury instead emits an average of about 260 tokens per multiple-choice answer, far more than the single letter the item asks for, and still finishes in about 1 second. Because we measure end-to-end latency, we cannot confirm the 1,107 tokens per second figure. We can only confirm that the speed shows up on our workload.

On par with Haiku and Luna, except on rare Javanese words

Inception's second claim puts Mercury 2.5 on par with GPT-5.6 Luna (Low) and Claude Haiku 4.5. On the two multiple-choice benchmarks that claim holds, with Mercury sitting between the two.

Indonesian slang Speech levels multiple choice

Claude Haiku 4.5

93.3
73.7

Mercury 2.5

91.7
81.6

GPT-5.6 Luna

100.0
89.5

The gap between Mercury and Claude Haiku 4.5 cannot be told apart from noise. On slang the two answer only 3 items differently, and on speech levels Mercury wins 6 items while Haiku wins 3 others. Against GPT-5.6 Luna, Mercury trails by 5 slang items, and even that gap is not quite enough to call real (p ≈ 0.06).

Rare Javanese words change the picture. Mercury answered 13 of 43 lemmas correctly, while GPT-5.6 Luna answered 35. We have not run Claude Haiku 4.5 on this set. Among the 16 models tested so far, including MiMo V2.6 Flash, which was run a week later, Mercury ranks 9th, 1 lemma behind GLM 5.3, and the 7 models below it score 11 or fewer.

Gemini 3.5 Flash

40

GPT-5.6 Luna

35

GPT-6 Astra

33

Gemma 4 31B

26

DeepSeek V4.1 Flash

22

DeepSeek V4 Flash

22

MiMo V2.6 Flash

15

GLM 5.3

14

Mercury 2.5

13

GLM 5.3 Flash

11

The gap to the models above it is not noise. Among the lemmas Mercury and Luna answered differently, Luna was right on 23 and Mercury on just 1 (p < 0.001). Against Gemma 4 31B the split is 14 to 1 (p ≈ 0.001), and against DeepSeek V4.1 Flash 10 to 1 (p ≈ 0.012). Rare vocabulary depends on knowledge already stored in the model, and assembling an answer faster adds none of it. Our data does not separate the diffusion approach from model size or training data, so we cannot yet name the exact cause.

DeepSeek V4.1 Flash: the same score at several times the cost

Against its predecessor, DeepSeek V4 Flash, DeepSeek V4.1 Flash gained 1 item on slang, 1 on speech levels, and nothing on rare Javanese words. Its cost is what moved.

Rare Javanese words

10.8

Indonesian slang

8.7

Speech levels multiple choice

6.7

The score gain is no more than a swap of items. On slang, V4.1 got 3 items right that its predecessor missed, but missed 2 that the predecessor had right. On rare words the two swapped exactly 1 lemma. All three gaps give p = 1.00. Output tokens rose too, from 4,981 to 21,239 on slang and from 6,495 to 10,803 on speech levels.

DeepSeek V4.1 Flash cost almost 11 times as much to get 22 rare Javanese words right, exactly as many as its predecessor.

Reasoning until it hits the cap

Every set has an answer token cap, and the cap on the rare Javanese words set is 600 tokens. On its first attempt, DeepSeek V4.1 Flash hit that cap on 17 of 43 lemmas.

Every time it hit the cap, no answer came out. The whole budget went to reasoning before a single word of answer was written. Our rules treat this as a technical failure, not a wrong answer, so we asked those lemmas again with a cap 4 times higher. At 2,400 tokens, 14 lemmas finished. The remaining 3 only got an answer at a 9,600-token cap, using 3,388 to 4,590 output tokens. They were Bladak, Kaleh and Sirtu, and 2 of those 3 answers were still wrong.

600 tokens

26

2,400 tokens

14

9,600 tokens

3

The time stretched with it. DeepSeek V4.1 Flash's median latency on rare Javanese words reached 8,310 ms, almost 2 times its predecessor's 4,670 ms. Its longest call took 218,641 ms for a single lemma, more than 3.5 minutes. We sent a reasoning budget of 128 tokens. On those 17 lemmas, DeepSeek V4.1 Flash spent 600 tokens without writing an answer, so the budget clearly did not apply.

A tie at the top of Javanese speech levels

On multiple-choice speech levels, DeepSeek V4.1 Flash answered 37 of 38 items correctly, the same as GPT-6 Astra and Gemini 3.5 Flash. Gemma 4 31B follows with 36. On the public slice, the only part shown on the benchmark page, all four scored 26 of 27.

That tie has one shared cause. The public item all four models missed is the same item, and not one of the 19 models in our panel at the time answered it correctly. The item text stays unpublished, like every item in this set. A tie at the top therefore does not mean DeepSeek V4.1 Flash beat the most expensive models in the panel. The public slice of this set can no longer separate the strongest models, the same pattern we found on the slang benchmark.

Price still separates them. One multiple-choice speech-level run cost $0.0131 for DeepSeek V4.1 Flash, $0.0725 for GPT-6 Astra, $0.0795 for Gemini 3.5 Flash, and $0.0006 for Gemma 4 31B. On open answers, the grader gave DeepSeek V4.1 Flash a full 8 of 8 and Mercury 2.5 7.75.

Limits of these results

These results do not show that diffusion models are weak at regional languages. We tested one diffusion model on one vocabulary set chosen for its rarity, while on the two other benchmarks Mercury 2.5 matched Claude Haiku 4.5. They do not show that DeepSeek V4.1 Flash is worse than its predecessor either, since its scores tie on all three tasks. We did not test coding, mathematical reasoning, or agentic tasks that may be the reason for this update.

The 1,107 tokens per second figure is also not confirmed by our data, because we measured end-to-end latency, not output speed. Nor do the costs here hold in general. Mercury 2.5's price is still a launch discount, and DeepSeek V4.1 Flash's cost depends on how often a given task triggers long reasoning.