Quick summary
Taken from the open answer, built-in prompt board, the board with the most models in this set. The other boards have their own figures below.
Results
How many questions each model answered correctly, from the exact same set. The bars show the count of correct answers, not a general ranking of model ability.
This set was run with 3 different instructions
The items and the answer key are identical across all 3 runs, 69 open answer items that do not move by a single word. Only the instruction sent to the model differs. Gemma 4 31B moves 44.9 points purely from changing the instruction. The winner changes too, from Qwen3 30B A3B Instruct 2507 to Gemma 4 31B. So this set has no single leaderboard, and the numbers must not be averaged across instructions.
What each instruction says
The prompt reads “Rewrite the sentence in standard Indonesian”, taken as it stands from inside the item. It is loose: it never says what must be left alone, so a model is free to read it as licence to swap words for synonyms.
open-py1The prompt forbids swapping words for synonyms and forbids changing the level of formality. That sounds reasonable, but our own answer key swaps words in 40 places, so this prompt contradicts the key it is scored against.
open-py2The prompt asks for every non-standard word to be made standard, including abbreviations written without vowels, and for already-standard words to be left alone. It was written after the key's rule was counted word by word.
open-py3Show 3 separate boards, one per instruction
The model writes its own answer, with no options offered. Some are scored automatically against a reference, some are read one by one by a native speaker using a weighted rubric.
A model whose name does not end in a date is called through an id that does not pin the version. The provider may swap the model behind that id at any time without notice, so the figures apply to whichever version was active when the run happened.
How these figures are computed
open-py1KeteranganThe prompt reads “Rewrite the sentence in standard Indonesian”, taken as it stands from inside the item. It is loose: it never says what must be left alone, so a model is free to read it as licence to swap words for synonyms.scoring open-py1KeteranganHow open answers are scored. The answer is normalised first, then compared with the reference and the list of other answers we accept. Items that require human judgement are deliberately left unscored.question version 5ca9dfe11dc2KeteranganA 12-character fingerprint of every question the model saw: item number, question text, answer options, and option order. Changing a single word changes the fingerprint. Scores under different fingerprints answer different questions, so this page only joins figures that share one.answer key version eda1d1cddc75KeteranganA 12-character fingerprint of the answer key: the reference, the other answers we accept, and speaker verdicts for items no machine may score. The questions can stay identical while the key changes, and when that happens scores move without a single word of the question changing. That is why it is kept apart from the question version.The model writes its own answer, with no options offered. Some are scored automatically against a reference, some are read one by one by a native speaker using a weighted rubric.
A model whose name does not end in a date is called through an id that does not pin the version. The provider may swap the model behind that id at any time without notice, so the figures apply to whichever version was active when the run happened.
How these figures are computed
open-py2KeteranganThe prompt forbids swapping words for synonyms and forbids changing the level of formality. That sounds reasonable, but our own answer key swaps words in 40 places, so this prompt contradicts the key it is scored against.scoring open-py1KeteranganHow open answers are scored. The answer is normalised first, then compared with the reference and the list of other answers we accept. Items that require human judgement are deliberately left unscored.question version 5ca9dfe11dc2KeteranganA 12-character fingerprint of every question the model saw: item number, question text, answer options, and option order. Changing a single word changes the fingerprint. Scores under different fingerprints answer different questions, so this page only joins figures that share one.answer key version eda1d1cddc75KeteranganA 12-character fingerprint of the answer key: the reference, the other answers we accept, and speaker verdicts for items no machine may score. The questions can stay identical while the key changes, and when that happens scores move without a single word of the question changing. That is why it is kept apart from the question version.The model writes its own answer, with no options offered. Some are scored automatically against a reference, some are read one by one by a native speaker using a weighted rubric.
A model whose name does not end in a date is called through an id that does not pin the version. The provider may swap the model behind that id at any time without notice, so the figures apply to whichever version was active when the run happened.
How these figures are computed
open-py3KeteranganThe prompt asks for every non-standard word to be made standard, including abbreviations written without vowels, and for already-standard words to be left alone. It was written after the key's rule was counted word by word.scoring open-py1KeteranganHow open answers are scored. The answer is normalised first, then compared with the reference and the list of other answers we accept. Items that require human judgement are deliberately left unscored.question version 5ca9dfe11dc2KeteranganA 12-character fingerprint of every question the model saw: item number, question text, answer options, and option order. Changing a single word changes the fingerprint. Scores under different fingerprints answer different questions, so this page only joins figures that share one.answer key version eda1d1cddc75KeteranganA 12-character fingerprint of the answer key: the reference, the other answers we accept, and speaker verdicts for items no machine may score. The questions can stay identical while the key changes, and when that happens scores move without a single word of the question changing. That is why it is kept apart from the question version.Per-model detail
The same figures as the chart above, plus the cost per correct answer, the wait time, and the date of the last run. Any column that cannot be guessed from its name carries its own explanation.
Show 3 separate tables, one per instruction
| Model | CorrectKeteranganHow many answers matched the key, out of the items scored. | Held backKeteranganThe score on items whose key we have never published. The column to its left covers items whose key is public, so a score there cannot be told apart from a model that read it. This column is what tells them apart. The held-back slice on this set is too small to read as a measurement: a gap only means something above roughly 34.9 points, far larger than any realistic gap. The percentage is therefore deliberately not shown, only the count. Publishing this score also wears it down over time, because every figure released is information about the items we hold back. That is why the held-back slice has a limited lifespan, and will be rotated. | $/correctKeteranganThe cost of one full run divided by ALL its correct answers, including those in the Held back column. The Correct column shows the public slice only, so the denominator here is the sum of both columns. NOT the cost per call: a cheap model that gets a lot wrong can cost more per correct answer than a model that costs more per call. | LatencyKeteranganMedian wait for a single call. Median rather than mean, so one stalled call does not move the number. | Last runKeteranganThe last date we ran this model on the same questions and answer key. |
|---|---|---|---|---|---|
| Qwen3 30B A3B Instruct 2507Alibaba | 29/4959.2% | 11/20 | $0.000024$0.0010 | 2,043 msmedian | |
| Ling-3.0-flashAnt Group | 27/4955.1% | 12/20 | $0.000050$0.0020 | 1,766 msmedian | |
| Mistral Small 3Mistral | 26/4953.1% | 13/20 | $0.000014$0.0005 | 962 msmedian | |
| DeepSeek V4 Flash 0731DeepSeek | 24/4949.0% | 11/20 | $0.000116$0.0040 | 3,644 msmedian | |
| Gemma-SEA-LION v4.5 E2B-ITlocal | 23/4946.9% | 12/20 | - | 890 msmedian | |
| GPT-5.6 LunaOpenAI | 22/4944.9% | 9/20 | $0.000069$0.0021 | 1,034 msmedian | |
| Gemma 4 31BGoogle | 21/4942.9% | 11/20 | $0.000051$0.0016 | 1,330 msmedian | |
| Solar Pro 4Upstage | 19/4938.8% | 9/20 | $0.000017$0.0005 | 2,684 msmedian | |
| Llama3 8B CPT Sahabat-AI v1 Instructlocal | 17/4934.7% | 5/20 | - | 390 msmedian | |
| Llama 3.1 8B Instructlocal | 16/4932.7% | 10/20 | - | 528 msmedian | |
| Hunyuan A13B InstructTencent | 12/4924.5% | 5/20 | $0.000114$0.0019 | 1,485 msmedian |
Models marked local run on our own hardware, so there is no bill to record and the wait time measures our machine, not a service anyone else can buy. The score is still comparable: the items, the key, and the temperature are identical to every other row here.
| Model | CorrectKeteranganHow many answers matched the key, out of the items scored. | Held backKeteranganThe score on items whose key we have never published. The column to its left covers items whose key is public, so a score there cannot be told apart from a model that read it. This column is what tells them apart. The held-back slice on this set is too small to read as a measurement: a gap only means something above roughly 34.9 points, far larger than any realistic gap. The percentage is therefore deliberately not shown, only the count. Publishing this score also wears it down over time, because every figure released is information about the items we hold back. That is why the held-back slice has a limited lifespan, and will be rotated. | $/correctKeteranganThe cost of one full run divided by ALL its correct answers, including those in the Held back column. The Correct column shows the public slice only, so the denominator here is the sum of both columns. NOT the cost per call: a cheap model that gets a lot wrong can cost more per correct answer than a model that costs more per call. | LatencyKeteranganMedian wait for a single call. Median rather than mean, so one stalled call does not move the number. | Last runKeteranganThe last date we ran this model on the same questions and answer key. |
|---|---|---|---|---|---|
| Gemma 4 31BGoogle | 43/4987.8% | 16/20 | $0.000026$0.0016 | 1,373 msmedian | |
| GPT-5.6 LunaOpenAI | 38/4977.6% | 14/20 | $0.000081$0.0042 | 2,199 msmedian | |
| Gemma-SEA-LION v4.5 E2B-ITlocal | 36/4973.5% | 15/20 | - | 863 msmedian | |
| DeepSeek V4 Flash 0731DeepSeek | 35/4971.4% | 15/20 | $0.000095$0.0047 | 4,526 msmedian | |
| Ling-3.0-flashAnt Group | 32/4965.3% | 12/20 | $0.000040$0.0018 | 1,767 msmedian | |
| Qwen3 30B A3B Instruct 2507Alibaba | 30/4961.2% | 13/20 | $0.000027$0.0012 | 1,656 msmedian | |
| Llama3 8B CPT Sahabat-AI v1 Instructlocal | 29/4959.2% | 11/20 | - | 415 msmedian | |
| Solar Pro 4Upstage | 28/4957.1% | 13/20 | $0.000013$0.0005 | 5,339 msmedian | |
| Mistral Small 3Mistral | 27/4955.1% | 11/20 | $0.000017$0.0006 | 650 msmedian | |
| Llama 3.1 8B Instructlocal | 20/4940.8% | 10/20 | - | 505 msmedian | |
| Hunyuan A13B InstructTencent | 19/4938.8% | 10/20 | $0.000078$0.0023 | 1,493 msmedian |
Models marked local run on our own hardware, so there is no bill to record and the wait time measures our machine, not a service anyone else can buy. The score is still comparable: the items, the key, and the temperature are identical to every other row here.
| Model | CorrectKeteranganHow many answers matched the key, out of the items scored. | Held backKeteranganThe score on items whose key we have never published. The column to its left covers items whose key is public, so a score there cannot be told apart from a model that read it. This column is what tells them apart. The held-back slice on this set is too small to read as a measurement: a gap only means something above roughly 34.9 points, far larger than any realistic gap. The percentage is therefore deliberately not shown, only the count. Publishing this score also wears it down over time, because every figure released is information about the items we hold back. That is why the held-back slice has a limited lifespan, and will be rotated. | $/correctKeteranganThe cost of one full run divided by ALL its correct answers, including those in the Held back column. The Correct column shows the public slice only, so the denominator here is the sum of both columns. NOT the cost per call: a cheap model that gets a lot wrong can cost more per correct answer than a model that costs more per call. | LatencyKeteranganMedian wait for a single call. Median rather than mean, so one stalled call does not move the number. | Last runKeteranganThe last date we ran this model on the same questions and answer key. |
|---|---|---|---|---|---|
| Gemma 4 31BGoogle | 43/4987.8% | 17/20 | $0.000027$0.0016 | 952 msmedian | |
| GPT-5.6 LunaOpenAI | 41/4983.7% | 14/20 | $0.000074$0.0041 | 2,020 msmedian | |
| DeepSeek V4 Flash 0731DeepSeek | 37/4975.5% | 14/20 | $0.000096$0.0049 | 4,094 msmedian | |
| Gemma-SEA-LION v4.5 E2B-ITlocal | 35/4971.4% | 15/20 | - | 930 msmedian | |
| Qwen3 30B A3B Instruct 2507Alibaba | 31/4963.3% | 15/20 | $0.000028$0.0013 | 1,636 msmedian | |
| Solar Pro 4Upstage | 30/4961.2% | 13/20 | $0.000012$0.0005 | 2,275 msmedian | |
| Ling-3.0-flashAnt Group | 29/4959.2% | 12/20 | $0.000083$0.0034 | 1,936 msmedian | |
| Mistral Small 3Mistral | 23/4946.9% | 15/20 | $0.000016$0.0006 | 671 msmedian | |
| Llama3 8B CPT Sahabat-AI v1 Instructlocal | 20/4940.8% | 9/20 | - | 418 msmedian | |
| Llama 3.1 8B Instructlocal | 12/4924.5% | 11/20 | - | 534 msmedian | |
| Hunyuan A13B InstructTencent | 11/4922.4% | 11/20 | $0.000099$0.0022 | 1,497 msmedian |
Models marked local run on our own hardware, so there is no bill to record and the wait time measures our machine, not a service anyone else can buy. The score is still comparable: the items, the key, and the temperature are identical to every other row here.
Every figure in this table can be recomputed from the raw data. Open the data →
Cost and tokensopen answerBuilt-in promptCheapest per correct answer: Mistral Small 3 ($0.000014). Most expensive: DeepSeek V4 Flash 0731 ($0.000116).Show chartsHide
All cost figures are in US dollars.
Cost per correct answer · Built-in prompt
The model with the highest score is not always the cheapest one. This divides the cost of one full test by its number of correct answers.
Cost of one full run · Built-in prompt
The raw figure, before dividing by the number of correct answers. Every model answered the exact same questions, so this is directly comparable: it is what you pay to run this benchmark once on each model.
Tokens used · Built-in prompt
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where almost all of the cost difference between models comes from. A model with no bar has no token record, which is not the same as zero.
Cost and tokensopen answerDetailed prompt forbidding word swapsCheapest per correct answer: Solar Pro 4 ($0.000013). Most expensive: DeepSeek V4 Flash 0731 ($0.000095).Show chartsHide
All cost figures are in US dollars.
Cost per correct answer · Detailed prompt forbidding word swaps
The model with the highest score is not always the cheapest one. This divides the cost of one full test by its number of correct answers.
Cost of one full run · Detailed prompt forbidding word swaps
The raw figure, before dividing by the number of correct answers. Every model answered the exact same questions, so this is directly comparable: it is what you pay to run this benchmark once on each model.
Tokens used · Detailed prompt forbidding word swaps
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where almost all of the cost difference between models comes from. A model with no bar has no token record, which is not the same as zero.
Cost and tokensopen answerPrompt stating the key's ruleCheapest per correct answer: Solar Pro 4 ($0.000012). Most expensive: Hunyuan A13B Instruct ($0.000099).Show chartsHide
All cost figures are in US dollars.
Cost per correct answer · Prompt stating the key's rule
The model with the highest score is not always the cheapest one. This divides the cost of one full test by its number of correct answers.
Cost of one full run · Prompt stating the key's rule
The raw figure, before dividing by the number of correct answers. Every model answered the exact same questions, so this is directly comparable: it is what you pay to run this benchmark once on each model.
Tokens used · Prompt stating the key's rule
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where almost all of the cost difference between models comes from. A model with no bar has no token record, which is not the same as zero.