Benchmarks tested

of 6 benchmarks we publish

This model has been run on every benchmark we publish.

Runs completed

complete runs on record

One run means the model answered every item in a set, start to finish. A set run more than once contributes more than one.

Total cost

$0.0000

for every run above

The real amount billed by the provider, summed, not an estimate from per-token pricing. Not yet divided by correct answers.

Total tokens

85,806

input and output combined

Counted only from runs that recorded tokens. Runs on the older path did not store them, and the figure is not guessed.

How many answers were correct

Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.

9 benchmarks

Pemahaman slang Indonesia32/42 76.2%

Koreksi ejaan Indonesia19/38 50.0%

Normalisasi teks Indonesiaquestion version 220/49 40.8%

Tingkat tutur bahasa Jawamultiple choice9/27 33.3%

Normalisasi teks Indonesiaquestion version 116/49 32.7%

Normalisasi teks Indonesiaquestion version 312/49 24.5%

Arti kata Jawa sehari-hari26/135 19.3%

Tingkat tutur bahasa Jawaopen answer1/6 16.7%

Arti kata Jawa langka1/43 2.3%

Share of items answered correctly across 9 benchmarks

How far the cost pays off

This model runs on our own hardware, not through a paid provider, so there is no bill to record. A zero here would make it the cheapest model we have ever tested, and that is not a figure we measured.

Tokens used

Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.

78,726 in, 7,080 out

inputoutput

Pemahaman slang Indonesia12,771 · 120

Koreksi ejaan Indonesia12,831 · 255

Normalisasi teks Indonesiaquestion version 213,098 · 955

Tingkat tutur bahasa Jawamultiple choice5,740 · 77

Normalisasi teks Indonesiaquestion version 110,752 · 945

Normalisasi teks Indonesiaquestion version 312,477 · 971

Arti kata Jawa sehari-hari8,062 · 2,910

Tingkat tutur bahasa Jawaopen answer1,231 · 103

Arti kata Jawa langka1,764 · 744

Tokens for one full run, per benchmark

Wait time per call

We do record its wait time, but it measures our own machine, not a service anyone else can buy. Putting it on the same chart as hosted models compares two different things, so the figure is not drawn here.

Test coverage

6/ 6

This model has been tested on all 6 benchmarks we publish.

6 of 6 benchmarks we publish

Earlier question versions

The questions or the key changed after these were measured. Kept as history, not as current results.show 2 rows

Pemahaman slang Indonesia22/27 81.5%

Arti kata Jawa sehari-hari26/135 19.3%

Results on an earlier version of the questions or answer key

Log11 calls to this model, spread across 3 days.Open

This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.

17 Aug 20261 runs

22:01Pemahaman slang Indonesiamultiple choice32/42

16 Aug 20261 runs

22:11Arti kata Jawa sehari-hariword meaning26/135

15 Aug 20269 runs

16:40Tingkat tutur bahasa Jawaopen answer1/6
16:40Arti kata Jawa sehari-hariword meaning26/135
16:36Arti kata Jawa langkaword meaning1/43
16:34Tingkat tutur bahasa Jawamultiple choice9/27
16:33Normalisasi teks Indonesiaopen answerPrompt stating the key's rule12/49
16:31Normalisasi teks Indonesiaopen answerDetailed prompt forbidding word swaps20/49
16:29Normalisasi teks Indonesiaopen answerBuilt-in prompt16/49
16:27Koreksi ejaan Indonesiaopen answer19/38
16:26Pemahaman slang Indonesiamultiple choice22/27

← See all models