Modellocal

Llama3 8B CPT Sahabat-AI v1 Instruct

How Llama3 8B CPT Sahabat-AI v1 Instruct performed across every Indonesian and regional-language benchmark we publish.Read more
Last runon Pemahaman slang Indonesia

We run this model on our own hardware, not through a paid provider. Its score is still comparable to other models, since the items, key, and temperature are identical, but its cost and wait time are not, so neither is drawn as a chart.

Benchmarks tested

6

of 6 benchmarks we publish

This model has been run on every benchmark we publish.

Runs completed

12

complete runs on record

One run means the model answered every item in a set, start to finish. A set run more than once contributes more than one.

Total cost

$0.0000

for every run above

The real amount billed by the provider, summed, not an estimate from per-token pricing. Not yet divided by correct answers.

Total tokens

99,322

input and output combined

Counted only from runs that recorded tokens. Runs on the older path did not store them, and the figure is not guessed.

How many answers were correct

Each bar is one benchmark: how many items it answered correctly out of every item in that set. We wrote the questions and answer keys ourselves and had them checked by two native speakers, and every model answers exactly the same items.

9 benchmarks1 run more than once
Pemahaman slang Indonesia2 runsKeterangan40-40/42 95.2%
Normalisasi teks Indonesiaquestion version 2Keterangan29/49 59.2%
Tingkat tutur bahasa Jawamultiple choice13/27 48.1%
Normalisasi teks Indonesiaquestion version 3Keterangan20/49 40.8%
Normalisasi teks Indonesiaquestion version 1Keterangan17/49 34.7%
Tingkat tutur bahasa Jawaopen answer1/6 16.7%
spread across runsbenchmarks without a dark block were run once
Share of items answered correctly across 9 benchmarks

How far the cost pays off

This model runs on our own hardware, not through a paid provider, so there is no bill to record. A zero here would make it the cheapest model we have ever tested, and that is not a figure we measured.

Tokens used

Input and output are kept apart because they are priced differently, often tenfold. A long output bar means the model talks a lot, and that is where nearly all the cost difference comes from.

91,497 in, 7,825 out
inputoutput
Pemahaman slang Indonesia2 runsKeterangan12,771 · 120
Normalisasi teks Indonesiaquestion version 2Keterangan13,098 · 939
Koreksi ejaan Indonesia12,831 · 205
Tingkat tutur bahasa Jawamultiple choice5,740 · 76
Arti kata Jawa sehari-hari8,062 · 3,281
Normalisasi teks Indonesiaquestion version 3Keterangan12,477 · 942
Normalisasi teks Indonesiaquestion version 1Keterangan10,752 · 950
Tingkat tutur bahasa Jawaopen answer1,231 · 132
Arti kata Jawa langka1,764 · 1,060
Tokens for one full run, per benchmark

Wait time per call

We do record its wait time, but it measures our own machine, not a service anyone else can buy. Putting it on the same chart as hosted models compares two different things, so the figure is not drawn here.

Test coverage

This model has been tested on all 6 benchmarks we publish.

6 of 6 benchmarks we publish

Earlier question versions

The questions or the key changed after these were measured. Kept as history, not as current results.
Pemahaman slang Indonesia22/27 81.5%
Arti kata Jawa sehari-hari62/135 45.9%
Results on an earlier version of the questions or answer key
Log12 calls to this model, spread across 4 days.

This model is called through a provider id that may not pin a version, so the dates below state which weights were actually tested. Rows marked with a hyphen have a date we inferred from file history rather than recorded at call time, so we do not show a clock time for them.

2 runs
  1. 22:10Pemahaman slang Indonesiamultiple choice40/42
  2. 22:04Pemahaman slang Indonesiamultiple choice40/42
1 runs
  1. 22:08Arti kata Jawa sehari-hariword meaning61/135
4 runs
  1. 00:56Arti kata Jawa sehari-hariword meaning62/135
  2. 00:52Arti kata Jawa langkaword meaning4/43
  3. 00:51Tingkat tutur bahasa Jawaopen answer1/6
  4. 00:50Tingkat tutur bahasa Jawamultiple choice13/27
5 runs
  1. 23:26Normalisasi teks Indonesiaopen answerPrompt stating the key's rule20/49
  2. 23:24Normalisasi teks Indonesiaopen answerDetailed prompt forbidding word swaps29/49
  3. 23:22Normalisasi teks Indonesiaopen answerBuilt-in prompt17/49
  4. 23:20Koreksi ejaan Indonesiaopen answer21/38
  5. 23:06Pemahaman slang Indonesiamultiple choice22/27

← See all models