How many answers were correct
Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.
9 benchmarks
Tingkat tutur bahasa Jawaopen answer6/6 100.0%
Pemahaman slang Indonesia41/42 97.6%
Tingkat tutur bahasa Jawamultiple choice26/27 96.3%
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.43/49 87.8%
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.43/49 87.8%
Arti kata Jawa sehari-hari116/135 85.9%
Koreksi ejaan Indonesia29/38 76.3%
Arti kata Jawa langka26/43 60.5%
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.21/49 42.9%
How far the cost pays off
Sorted by the cheapest correct answer first. The left column shows how many items were answered correctly, so a top row with a long left bar means both cheap and accurate.
9 with cost on record
Correct answersCost per correct answer
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.87.8%$0.000026
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.87.8%$0.000027
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.42.9%$0.000051
Tokens used
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.
62,782 in, 6,188 out
inputoutput
Tingkat tutur bahasa Jawaopen answer962 · 97
Pemahaman slang Indonesia10,271 · 120
Tingkat tutur bahasa Jawamultiple choice4,908 · 76
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.9,761 · 901
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.10,432 · 688
Arti kata Jawa sehari-hari7,027 · 3,431
Koreksi ejaan Indonesia10,323 · 190
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.9,098 · 685
1 benchmarks are not drawn here: those runs came from an older path whose ledger did not store token counts. Not recorded, not zero.
Wait time per call
Median wait for a single call on each benchmark. Median rather than mean, so one stalled call does not move the number.
9 with wait time on record
Tingkat tutur bahasa Jawamultiple choice0.80 s
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.0.95 s
Koreksi ejaan Indonesia1.02 s
Tingkat tutur bahasa Jawaopen answer1.32 s
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.1.33 s
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.1.37 s
Arti kata Jawa langka1.91 s
Test coverage
6/ 6
This model has been tested on all 6 benchmarks we publish.
Earlier question versions
The questions or the key changed after these were measured. Kept as history, not as current results.show 2 rowsclose
Pemahaman slang Indonesia27/27 100.0%
Arti kata Jawa sehari-hari3 runsKeteranganThis benchmark was run 3 times on this version of the items or key, and the bar draws the midpoint of those 3 results.116/135 84.8%
Log13 calls to this model, spread across 5 days.OpenClose
This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.
1 runs
- 22:43Pemahaman slang Indonesiamultiple choice41/42
1 runs
- 20:35Arti kata Jawa sehari-hariword meaning116/135
2 runs
- -Keteranganestimated dateArti kata Jawa sehari-hariword meaning113/135
- 06:01Tingkat tutur bahasa Jawaopen answer6/6
7 runs
- 22:43Tingkat tutur bahasa Jawamultiple choice26/27
- 18:08Normalisasi teks Indonesiaopen answerPrompt stating the key's rule43/49
- 16:01Normalisasi teks Indonesiaopen answerDetailed prompt forbidding word swaps43/49
- 15:06Normalisasi teks Indonesiaopen answerBuilt-in prompt21/49
- 14:58Koreksi ejaan Indonesiaopen answer29/38
- 14:51Pemahaman slang Indonesiamultiple choice27/27
- 00:10Arti kata Jawa sehari-hariword meaning116/135
2 runs
- 22:57Arti kata Jawa sehari-hariword meaning115/135
- 20:46Arti kata Jawa langkaword meaning26/43