This set has 1 other set of results, from earlier versions of the questions or the answer key. We do not draw them on the same chart, because a score computed against different questions is not the same measurement.
Quick summary
Results
How many questions each model answered correctly, from the exact same set. The bars show the count of correct answers, not a general ranking of model ability.
The model reads the question with its answer options, then replies with a single letter. Scoring is automatic: correct when the letter matches the key.
A model whose name does not end in a date is called through an id that does not pin the version. The provider may swap the model behind that id at any time without notice, so the figures apply to whichever version was active when the run happened.
How these figures are computed
mcq-py1KeteranganHow multiple choice is asked. Question and options are built from the item, option order is shuffled with a fixed seed so every model sees the same order, and the model is asked for a single letter.scoring mcq-py1KeteranganHow multiple choice is scored. We read one option letter from the answer and compare it with the key. An answer with no option letter is recorded as a format failure, kept apart from a wrong answer, so a broken harness does not read as a weak model.question version d3e8c01bc554KeteranganA 12-character fingerprint of every question the model saw: item number, question text, answer options, and option order. Changing a single word changes the fingerprint. Scores under different fingerprints answer different questions, so this page only joins figures that share one.answer key version 4e94e1cb7f25KeteranganA 12-character fingerprint of the answer key: the reference, the other answers we accept, and speaker verdicts for items no machine may score. The questions can stay identical while the key changes, and when that happens scores move without a single word of the question changing. That is why it is kept apart from the question version.Per-model detail
The same figures as the chart above, plus the cost per correct answer, the wait time, and the date of the last run. Any column that cannot be guessed from its name carries its own explanation.
| Model | CorrectKeteranganHow many answers matched the key, out of the items scored. | Held backKeteranganThe score on items whose key we have never published. The column to its left covers items whose key is public, so a score there cannot be told apart from a model that read it. This column is what tells them apart. The held-back slice on this set is too small to read as a measurement: a gap only means something above roughly 36.8 points, far larger than any realistic gap. The percentage is therefore deliberately not shown, only the count. Publishing this score also wears it down over time, because every figure released is information about the items we hold back. That is why the held-back slice has a limited lifespan, and will be rotated. | $/correctKeteranganThe cost of one full run divided by ALL its correct answers, including those in the Held back column. The Correct column shows the public slice only, so the denominator here is the sum of both columns. NOT the cost per call: a cheap model that gets a lot wrong can cost more per correct answer than a model that costs more per call. | LatencyKeteranganMedian wait for a single call. Median rather than mean, so one stalled call does not move the number. | Last runKeteranganThe last date we ran this model on the same questions and answer key. |
|---|---|---|---|---|---|
| GPT-5.6 LunaOpenAI | 42/42100.0% | 18/18 | $0.000051$0.0031 | 1,103 msmedian | |
| Gemma 4 31BGoogle | 41/4297.6% | 17/18 | $0.000017$0.0010 | 614 msmedian | |
| Llama3 8B CPT Sahabat-AI v1 Instructlocal2 runsKeteranganThis model was run 2 times on this benchmark, with the same items, key, and temperature. The Correct column shows the lowest through the highest result. That distance shows how far the score moves on its own, so any difference smaller than it is not a difference in ability. Cost, latency, and date come from the most recent run rather than being summed. | 56-56/4295.2% | 16/18 | - | 430 msmedian | |
| DeepSeek V4 Flash 0731DeepSeek | 39/4292.9% | 18/18 | $0.000039$0.0022 | 3,281 msmedian | |
| Claude Haiku 4.5Anthropic | 38/4290.5% | 18/18 | $0.000274$0.0154 | 1,426 msmedian | |
| Ling-3.0-flashAnt Group | 37/4288.1% | 17/18 | $0.000021$0.0011 | 1,410 msmedian | |
| Qwen3 30B A3B Instruct 2507Alibaba | 37/4288.1% | 16/18 | $0.000021$0.0011 | 701 msmedian | |
| Solar Pro 4Upstage | 37/4288.1% | 15/18 | $0.000008$0.0004 | 881 msmedian | |
| Gemma-SEA-LION v4.5 E2B-ITlocal | 33/4278.6% | 12/18 | - | 843 msmedian | |
| Mistral Small 3Mistral | 32/4276.2% | 13/18 | $0.000013$0.0006 | 501 msmedian | |
| Llama 3.1 8B Instructlocal | 32/4276.2% | 13/18 | - | 533 msmedian | |
| Hunyuan A13B InstructTencent | 29/4269.0% | 14/18 | $0.000042$0.0018 | 1,488 msmedian |
Models marked local run on our own hardware, so there is no bill to record and the wait time measures our machine, not a service anyone else can buy. The score is still comparable: the items, the key, and the temperature are identical to every other row here.
Every figure in this table can be recomputed from the raw data. Open the data →
Cost and tokensCheapest per correct answer: Solar Pro 4 ($0.000008). Most expensive: Claude Haiku 4.5 ($0.000274).Show chartsHide
All cost figures are in US dollars.
Cost per correct answer
The model with the highest score is not always the cheapest one. This divides the cost of one full test by its number of correct answers.
Cost of one full run
The raw figure, before dividing by the number of correct answers. Every model answered the exact same questions, so this is directly comparable: it is what you pay to run this benchmark once on each model.
Tokens used
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where almost all of the cost difference between models comes from. A model with no bar has no token record, which is not the same as zero.
Over time
Only models run more than once are drawn. One point is not a trend, and drawing it as a flat line implies a stability we have not measured.
| Deret | Tanggal | Nilai |
|---|---|---|
| Llama3 8B CPT Sahabat-AI v1 Instruct | 2026-08-17T15:04:03Z | 56 |
| Llama3 8B CPT Sahabat-AI v1 Instruct | 2026-08-17T15:10:43Z | 56 |
Earlier versions
The questions or the key changed after these numbers were measured, so they cannot be compared directly with the results above. We keep them rather than delete them.