This tiny AI model tried to Google our exam answers

The two lightest fighters ever in our instrument

After testing the mystery model ox-alpha this week, we dropped down to the lightest class, the featherweights. LFM2.5-2.6B, made by Liquid AI: 2.6 billion parameters, free, a hybrid architecture that is not a pure transformer, and built as a small on-device agent that always deliberates before answering. Its opponent, hy-mt2-1.8b, made by Tencent: 1.8 billion parameters, the cheapest rate we have ever paid, and it answers instantly without much thought. The arena: 60 multiple-choice questions on Indonesian slang comprehension from our benchmark.

As usual for models outside the panel: neither score is on the ibahasa.com/benchmark leaderboard, and neither ever will be. We measured both in the trial lane, a separate track that uses the same questions and scoring rubric as the official board but keeps the results quarantined.

How we ran the benchmark, and why there is no Javanese set

Our slang set holds 60 multiple-choice questions whose questions and answer key have never been published, so no model could have memorized them from training data. Each model answers once per question, answers are scored automatically against the key, and the whole episode cost less than a tenth of a cent.

Readers of our earlier articles will ask: where is the Javanese krama round? We deliberately left it out for this model. Free-tier models typically retain every prompt that comes in, while some of our questions are contamination-detection reserves whose entire value lies in staying secret. We chose not to leak those questions bit by bit into someone else's logs for the sake of one article. The Javanese round will follow once our reserve-question filter is finished.

The numbers

hy-mt2 (1.8B)

34

LFM2.5 (2.6B)

32

Two bars nearly the same length are the main finding. A two-answer gap out of 60 gives z around 0.4: a flat tie. Both are also well above the 25% random-guess floor, so this is not two models both guessing blindly. The featherweight bout ends in a draw, and what separates them is not the score.

One answers fast, one meditates (reasons) first

LFM2.5 (2.6B)

715

hy-mt2 (1.8B)

2

The difference in temperament shows up in the chart above. hy-mt2 answers with just the letter: an average of 2 tokens per question, a median of 467 milliseconds. LFM2.5 always switches on its "meditation," or reasoning, mode first: it averages 715 tokens per question, a median of 2.4 seconds, roughly 358 times the output of its opponent, to land on a score that is statistically the same. On lexical-knowledge questions like slang, that much of a thinking budget buys nothing: a word it does not know stays unknown, no matter how long it mulls it over. An honest caveat: the latency in the table is the latency of the service we used, not the on-device speed advertised by the makers. Token counts, by contrast, do not care whether the server was busy or idle.

The fighter that tried to google its answer

One question in our set uses a trap lemma: a word designed to test whether a model invents a meaning for a word it does not know. hy-mt2 answered it wrong calmly, one letter as usual. LFM2.5 did something we have never seen in this instrument before: instead of answering, it emitted a tool call, complete with two search queries written bilingually, roughly shaped like this, with the keyword itself masked:

<|tool_call_start|>[google(query='<lemma> bahasa Indonesia slang meaning'),

                    google(query='<lemma> kata gaul Indonesia')]<|tool_call_end|>

LFM2.5 was in fact trained as an agentic model with built-in tool calling (tool_calls), and inside a closed exam room with no tools available, that built-in habit fired in the wrong place: LFM2.5 did not know the answer, so it went off trying to find "divine guidance." That answer was recorded as-is and scored zero, because we grade the model as it is deployed. But as behavior, it is a small window into the new generation of models: today's tiny models no longer just pick the wrong letter, they can fail agentically.

We nearly got it wrong

First, we nearly wrote "this agentic model panics and googles things" as a pattern. Then we counted: the tool call appeared on only 1 of 60 questions. On the rest, LFM2.5 answered like an ordinary student. So we report this incident as a single-question anecdote, not a failure mode.

Second, our own instrument nearly shortchanged LFM2.5. One question failed because its answer was cut off at the 2,400-token limit, its habit of long deliberation did not fit. We reran that question with a 9,600-token limit, and it answered correctly. Without that recovery its score would read 31, not 32, and the gap would have come from our token limit, not from its ability. For a model that always meditates first, room to think is part of measuring it fairly.

Conclusion

A tie is a win for no one: 34 versus 32 out of 60 does not separate these two models. We measured each model only once, and only on multiple-choice Indonesian slang. So the honest conclusion stays that narrow: neither model's other abilities have been tested, and if the measurement were repeated, the numbers could shift by one or two answers. The two architectures are genuinely different, but this data is not enough to say which architecture suits Indonesian better. Only the price looks decisive: a 358-fold thinking budget for an equal result. And neither model has been tested yet on Javanese speech levels, the hardest part of our instrument, for the reason explained above.

The derived data is available in the trial-model-probes dataset linked below this article. It contains per-model summaries, including latency and tokens, so every claim here can be recomputed.