How many answers were correct
Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.
7 benchmarks
Pemahaman slang Indonesia37/42 88.1%
Tingkat tutur bahasa Jawa18/27 66.7%
Koreksi ejaan Indonesia25/38 65.8%
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.32/49 65.3%
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.29/49 59.2%
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.27/49 55.1%
Arti kata Jawa sehari-hari69/135 51.1%
How far the cost pays off
Sorted by the cheapest correct answer first. The left column shows how many items were answered correctly, so a top row with a long left bar means both cheap and accurate.
7 with cost on record
Correct answersCost per correct answer
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.65.3%$0.000040
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.55.1%$0.000050
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.59.2%$0.000083
Tokens used
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.
80,123 in, 158,140 out
inputoutput
Pemahaman slang Indonesia13,334 · 10,279
Tingkat tutur bahasa Jawa6,245 · 13,504
Koreksi ejaan Indonesia13,118 · 16,337
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.13,455 · 22,398
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.12,351 · 42,114
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.10,971 · 25,604
Arti kata Jawa sehari-hari10,649 · 27,904
Wait time per call
Median wait for a single call on each benchmark. Median rather than mean, so one stalled call does not move the number.
7 with wait time on record
Koreksi ejaan Indonesia1.41 s
Normalisasi teks Indonesiaquestion version 1KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.1.77 s
Normalisasi teks Indonesiaquestion version 2KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.1.77 s
Normalisasi teks Indonesiaquestion version 3KeteranganWe asked this benchmark with several different phrasings of the question, on the same items and the same answer key. Scores across phrasings are not directly comparable; open the benchmark page to see them side by side.1.94 s
Test coverage
5/ 6
This model has been tested on 5 of the 6 benchmarks we publish. The rest have not been run, so this page cannot say anything about how it does there.
Earlier question versions
The questions or the key changed after these were measured. Kept as history, not as current results.show 2 rowsclose
Pemahaman slang Indonesia25/27 92.6%
Arti kata Jawa sehari-hari69/135 51.1%
Log9 calls to this model, spread across 7 days.OpenClose
This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.
1 runs
- 03:19Normalisasi teks Indonesiaopen answerDetailed prompt forbidding word swaps32/49
1 runs
- 19:42Tingkat tutur bahasa Jawamultiple choice18/27
1 runs
- 22:39Pemahaman slang Indonesiamultiple choice37/42
1 runs
- 21:10Arti kata Jawa sehari-hariword meaning69/135
2 runs
- 06:47Koreksi ejaan Indonesiaopen answer25/38
- 06:46Pemahaman slang Indonesiamultiple choice25/27
2 runs
- 18:28Normalisasi teks Indonesiaopen answerPrompt stating the key's rule29/49
- 15:34Normalisasi teks Indonesiaopen answerBuilt-in prompt27/49
1 runs
- 22:22Arti kata Jawa sehari-hariword meaning69/135