How many answers were correct
Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.
How far the cost pays off
Sorted by the cheapest correct answer first. The left column shows how many items were answered correctly, so a top row with a long left bar means both cheap and accurate.
Tokens used
Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.
1 benchmarks are not drawn here: those runs came from an older path whose ledger did not store token counts. Not recorded, not zero.
Wait time per call
Median wait for a single call on each benchmark. Median rather than mean, so one stalled call does not move the number.
Test coverage
This model has been tested on 3 of the 6 benchmarks we publish. The rest have not been run, so this page cannot say anything about how it does there.
Earlier question versions
The questions or the key changed after these were measured. Kept as history, not as current results.
Log5 calls to this model, spread across 2 days.OpenClose
This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.
- 21:55Arti kata Jawa sehari-hariword meaning128/135
- -Keteranganestimated dateArti kata Jawa langkaword meaning40/43
- -Keteranganestimated dateArti kata Jawa sehari-hariword meaning127/135
- 06:02Tingkat tutur bahasa Jawaopen answer6/6
- 04:01Tingkat tutur bahasa Jawamultiple choice26/27