Benchmarks tested

of 6 benchmarks we publish

This model has been run on every benchmark we publish.

Runs completed

complete runs on record

One run means the model answered every item in a set, start to finish. A set run more than once contributes more than one.

Total cost

$0.0239

for every run above

The real amount billed by the provider, summed, not an estimate from per-token pricing. Not yet divided by correct answers.

Total tokens

116,192

input and output combined

Counted only from runs that recorded tokens. Runs on the older path did not store them, and the figure is not guessed.

How many answers were correct

Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.

7 benchmarks

Pemahaman slang Indonesia40/42 95.2%

Tingkat tutur bahasa Jawaopen answer5/6 83.3%

Tingkat tutur bahasa Jawamultiple choice22/27 81.5%

Normalisasi teks Indonesia37/49 75.5%

Arti kata Jawa sehari-hari100/135 74.1%

Koreksi ejaan Indonesia26/38 68.4%

Arti kata Jawa langka15/43 34.9%

Share of items answered correctly across 7 benchmarks

How far the cost pays off

Sorted by the cheapest correct answer first. The left column shows how many items were answered correctly, so a top row with a long left bar means both cheap and accurate.

7 with cost on record

Correct answersCost per correct answer

Arti kata Jawa sehari-hari74.1%$0.000045

Normalisasi teks Indonesia75.5%$0.000056

Pemahaman slang Indonesia95.2%$0.000056

Koreksi ejaan Indonesia68.4%$0.000116

Arti kata Jawa langka34.9%$0.000119

Tingkat tutur bahasa Jawaopen answer83.3%$0.000125

Tingkat tutur bahasa Jawamultiple choice81.5%$0.000143

Correct answers against cost per correct answer

Tokens used

Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.

51,619 in, 64,573 out

inputoutput

Pemahaman slang Indonesia12,564 · 5,778

Tingkat tutur bahasa Jawaopen answer1,203 · 2,168

Tingkat tutur bahasa Jawamultiple choice5,593 · 13,119

Normalisasi teks Indonesia10,479 · 7,950

Arti kata Jawa sehari-hari7,470 · 19,193

Koreksi ejaan Indonesia12,674 · 10,784

Arti kata Jawa langka1,636 · 5,581

Tokens for one full run, per benchmark

Wait time per call

Median wait for a single call on each benchmark. Median rather than mean, so one stalled call does not move the number.

7 with wait time on record

Pemahaman slang Indonesia2.31 s

Arti kata Jawa langka3.25 s

Arti kata Jawa sehari-hari3.27 s

Normalisasi teks Indonesia3.35 s

Koreksi ejaan Indonesia3.65 s

Tingkat tutur bahasa Jawamultiple choice5.55 s

Tingkat tutur bahasa Jawaopen answer7.02 s

Median wait for one call, per benchmark

Test coverage

6/ 6

This model has been tested on all 6 benchmarks we publish.

6 of 6 benchmarks we publish

Log7 calls to this model, spread across 1 days.Open

This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.

22 Sept 20267 runs

21:00Arti kata Jawa sehari-hariword meaning100/135
20:36Normalisasi teks Indonesiaopen answer37/49
20:26Koreksi ejaan Indonesiaopen answer26/38
20:15Tingkat tutur bahasa Jawaopen answer5/6
20:14Tingkat tutur bahasa Jawamultiple choice22/27
19:29Arti kata Jawa langkaword meaning15/43
19:27Pemahaman slang Indonesiamultiple choice40/42

See the whole panel's run log →

← See all models