Evaluating ox-alpha before Z.ai revealed it as GLM-5.3-Flash

The same model, two grading tables

In our previous article we measured ox-alpha, the viral stealth model, in our trial lane: a quarantined path that uses the same questions and graders as the main board but whose numbers never enter the rankings. We made two promises there. Ox-alpha will never appear on the main board, because its identity would change. And GLM-5.3, the official Z.ai model that looked like a half-sibling, would be run on the main board in early September.

Three days later, on August 26, Z.ai spoke: ox-alpha was GLM-5.3-Flash, the budget version of GLM-5.3. Both predictions ripened at once. The official model now sits in the paid catalog at $0.075 per million input tokens and $0.25 per million output tokens, so it may and must enter the main board with its withheld slice, closing the second promise.

We now hold something almost nobody else has: the same model, measured twice, before and after its identity went public. The question has changed shape. No longer "who is it", but: are the free model and its official version actually the same model at serving time?

What we compared

One row of that table deserves a longer story. Ox-alpha stored user prompts at the time, so we refused to send it the withheld items; that policy predates ox-alpha by weeks and is not a reaction to one model. Because of it, this comparison uses the public slice of both sets: 42 Indonesian slang items and 38 Javanese speech-level items. Every number in this article was recomputed from the ledgers of both sessions on the day of writing, not copied from old notes.

An unplanned coincidence: the two sessions sit only eight days apart, and the first ran two days before Z.ai's announcement. If, as the community suspected, the free promotion was used to harvest real-world usage, the training data it gathered had no time to move the weights of the official model we measured yesterday. That is speculation, and we flag it as such. But it makes this eight-day window about as clean a comparison as one can hope for without access to anyone's servers.

Indonesian slang and the multiple-choice Javanese items are scored by machine against the key. There is a third arm that does not appear in that chart, and it is the most interesting one.

The score rose on both sets

On Indonesian slang, ox-alpha answered 49 of 60 correctly when we measured it in the trial lane. Its official version, eight days later on the main board, answered 54 of 60. On Javanese speech levels, 27 of 38 rose to 32 of 38. Both sets moved the same direction by nearly the same margin: 5 answers, twice.

Of the 60 slang items, 53 were answered with the exact same letter by both models. Of the 38 Javanese items, 30. The answer match is 88% and 79%, up from the 71% we reported between ox-alpha and the full GLM-5.3 last time. The official version also appears closer to the stealth model than to its own bigger sibling.

But 5 out of 60, how big is that? On both sets, only 7 items were answered differently by the two models, and within those 7 the official version was right on 6 while ox-alpha was right on 1. An exact McNemar test, built precisely for paired comparisons like this, gives p around 0.125 on both sets. Two sets pointing the same way beat one set, but these numbers do not close the door on chance. What we can claim is not "the official version got smarter", but "the official version scored higher on both sets, with a consistent pattern".

The human-scored arm

The Javanese speech-level set has 8 open-ended questions a machine cannot score: the model is asked to rewrite ngoko sentences into krama alus, and a native speaker grades each answer against a weighted rubric. In the trial lane, our reviewer accepted ox-alpha at 6.72 out of 8. Its official version, graded with the same rubric by the same reviewer, earned 7.67 out of 8.

The gap is a full point, and in this arm it means more than 5 multiple-choice answers. Open-ended answers are judged on the quality of the language, not on a single letter, and the same rubric applied to both. One small difference we must write down: the trial-lane verdict was given before ox-alpha's identity went public, with a different grader flag from the official one, so the two must not be read as one shared column of scores.

Its withheld score stayed quiet: 2 of 2 answers on the withheld questions received a perfect score. No contamination signal changed between the two sessions, though 2 questions are far too few to close that possibility. This is why our staged measurement keeps running.

Why the free model and its official self differ

The GLM-5.3-Flash weights are open and downloadable, so in theory the two should answer identically. They do not, and the cause is no mystery: weights are one layer of a serving model. At least four layers can differ between two endpoints, and all four are documented elsewhere, not invented here:

1. Hidden system prompts. A stealth model never runs undressed. Independent forensics on ox-alpha found roughly 75 extra tokens appended to every request, most served from a cache with identical content each time, including instructions to keep its identity quiet. Its official self carries no such hiding instructions. Different instructions shift answers, however small the appended text.
2. Serving quantization. The weights you download and the weights a server runs are not always the same precision. Dropping from 16-bit to 8-bit or lower slashes serving cost, and small weight shifts are enough to flip a single unsure multiple-choice answer. Without a provider statement, two endpoints may serve different precisions.
3. Routing and sampling defaults. One id on an aggregator can route to different backends, with different default temperature or top_p. The practice is common enough that research on OpenRouter's free versus paid endpoints found statistically different behavior on the exact same prompts, with directions that were not even consistent across models.
4. Training contamination during the promotion. This is the question that opened the article. A free endpoint that stores prompts, plus a promotion built to attract real usage, is a combination that, if used for training, changes the weights. Z.ai's announcement mentioned a weight release, not their training practice, so this layer stays open. What we can say: an eight-day window is too short for one training cycle, so the score rise most likely comes from the first three layers.

All four share one consequence: a stealth model's score is the score of that serving configuration, not of its weights. If someone else measures ox-alpha this week on another endpoint, their number may differ from ours, and nobody is wrong. That is why stealth models never enter our main board, and the reason can now be shown in numbers: a shifting identity produces measurements nobody can reproduce. The human-scored arm above points the same way from the other side: ox-alpha 6.72 and its official self 7.67, a gap that would not vanish even with identical quantization or routing, because what moved there is the quality of the answer itself in a native speaker's eyes.

What these numbers cannot tell you

Three things must not be concluded from this article. First, that GLM-5.3-Flash "improved" since its promotion: one run per side, and p around 0.125 does not rule out chance. Second, that the gap has one specific cause: we hold no server access, so which layer moved cannot be separated from the outside. Third, that ox-alpha and its official self are weight-identical: an 88% match is high, but model identity can never be confirmed from language behavior alone, and Z.ai's official statement is the best evidence we hold, not our own verification.

What a reader can take away: if you evaluate a model through anyone's free endpoint, you are evaluating the service, not the model. Measure the official one, at its stated price, before you decide anything from it.