AnthropicModel

Claude Sonnet 5.5

How Claude Sonnet 5.5 performed across every Indonesian and regional-language benchmark we publish.Read more
Last runon Arti kata Jawa langka

Benchmarks tested

6

of 6 benchmarks we publish

This model has been run on every benchmark we publish.

Runs completed

9

complete runs on record

One run means the model answered every item in a set, start to finish. A set run more than once contributes more than one.

Total cost

$0.9639

for every run above

The real amount billed by the provider, summed, not an estimate from per-token pricing. Not yet divided by correct answers.

Total tokens

195,999

input and output combined

Counted only from runs that recorded tokens. Runs on the older path did not store them, and the figure is not guessed.

How many answers were correct

Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.

9 benchmarks
Tingkat tutur bahasa Jawaopen answer6/6 100.0%
Tingkat tutur bahasa Jawamultiple choice26/27 96.3%
Pemahaman slang Indonesia40/42 95.2%
Arti kata Jawa sehari-hari122/135 90.4%
Arti kata Jawa langka34/43 79.1%
Koreksi ejaan Indonesia28/38 73.7%
Normalisasi teks Indonesiaquestion version 3Keterangan32/49 65.3%
Normalisasi teks Indonesiaquestion version 2Keterangan31/49 63.3%
Normalisasi teks Indonesiaquestion version 1Keterangan20/49 40.8%
Share of items answered correctly across 9 benchmarks

How far the cost pays off

Sorted by the cheapest correct answer first. The left column shows how many items were answered correctly, so a top row with a long left bar means both cheap and accurate.

9 with cost on record
Correct answersCost per correct answer
Tingkat tutur bahasa Jawamultiple choice96.3%$0.001172
Pemahaman slang Indonesia95.2%$0.001301
Arti kata Jawa sehari-hari90.4%$0.001637
Tingkat tutur bahasa Jawaopen answer100.0%$0.001695
Normalisasi teks Indonesiaquestion version 2Keterangan63.3%$0.002089
Normalisasi teks Indonesiaquestion version 1Keterangan40.8%$0.002219
Koreksi ejaan Indonesia73.7%$0.003165
Normalisasi teks Indonesiaquestion version 3Keterangan65.3%$0.003167
Arti kata Jawa langka79.1%$0.003332
Correct answers against cost per correct answer

Tokens used

Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.

124,509 in, 71,490 out
inputoutput
Tingkat tutur bahasa Jawaopen answer1,941 · 968
Tingkat tutur bahasa Jawamultiple choice8,288 · 2,679
Pemahaman slang Indonesia19,318 · 3,550
Arti kata Jawa sehari-hari12,585 · 26,291
Arti kata Jawa langka2,743 · 10,779
Koreksi ejaan Indonesia20,756 · 8,827
Normalisasi teks Indonesiaquestion version 3Keterangan19,948 · 10,580
Normalisasi teks Indonesiaquestion version 2Keterangan21,328 · 5,345
Normalisasi teks Indonesiaquestion version 1Keterangan17,602 · 2,471
Tokens for one full run, per benchmark

Wait time per call

Median wait for a single call on each benchmark. Median rather than mean, so one stalled call does not move the number.

9 with wait time on record
Pemahaman slang Indonesia1.05 s
Normalisasi teks Indonesiaquestion version 1Keterangan1.09 s
Tingkat tutur bahasa Jawamultiple choice1.15 s
Normalisasi teks Indonesiaquestion version 2Keterangan1.17 s
Arti kata Jawa sehari-hari1.48 s
Normalisasi teks Indonesiaquestion version 3Keterangan1.78 s
Koreksi ejaan Indonesia2.72 s
Tingkat tutur bahasa Jawaopen answer3.30 s
Arti kata Jawa langka3.56 s
Median wait for one call, per benchmark

Test coverage

This model has been tested on all 6 benchmarks we publish.

6 of 6 benchmarks we publish
Log9 calls to this model, spread across 1 days.

This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.

9 runs
  1. 07:49Arti kata Jawa langkaword meaning34/43
  2. 07:45Arti kata Jawa sehari-hariword meaning122/135
  3. 07:31Normalisasi teks Indonesiaopen answerPrompt stating the key's rule32/49
  4. 07:25Normalisasi teks Indonesiaopen answerDetailed prompt forbidding word swaps31/49
  5. 07:19Normalisasi teks Indonesiaopen answerBuilt-in prompt20/49
  6. 07:15Koreksi ejaan Indonesiaopen answer28/38
  7. 07:09Tingkat tutur bahasa Jawaopen answer6/6
  8. 07:08Tingkat tutur bahasa Jawamultiple choice26/27
  9. 07:05Pemahaman slang Indonesiamultiple choice40/42

← See all models