Indonesian Language Observatory
Benchmark Nusantara
We test AI models on Indonesian and its regional languages ourselves, using questions and answer keys we wrote and had checked by two native speakers. The numbers are published as they are, limits included.Read more
6 benchmarks471 itemsLast tested
Indonesia
Indonesian spelling correction
Accepting or rejecting spell-checker suggestions on real text.
54 items16 held back16 modelsLast tested
Score spread
Gemma 4 31B29/38 76.3%MiMo-V2.6-Flash26/38 68.4%GLM 5.326/38 68.4%GLM 5.3 Flash25/38 65.8%Ling-3.0-flash25/38 65.8%GPT-5.6 Luna25/38 65.8%Gemma-SEA-LION v4.5 E2B-IT25/38 65.8%DeepSeek V4 Flash 073123/38 60.5%Solar Pro 423/38 60.5%Seed-2.0-Mini22/38 57.9%Mistral Small 322/38 57.9%Qwen3 30B A3B Instruct 250721/38 55.3%Llama3 8B CPT Sahabat-AI v1 Instruct21/38 55.3%Hunyuan A13B Instruct21/38 55.3%Claude Haiku 4.520/38 52.6%Llama 3.1 8B Instruct19/38 50.0%
Gemma 4 31B29/38 76.3%
Indonesia
Indonesian text normalisation
Turning text-message style writing into standard Indonesian.
69 items20 held back15 modelsLast tested
Score spread
open answer, range across 3 prompts15 models
MiMo-V2.6-Flash76–76%across promptsGLM 5.361–76%across promptsGemma 4 31B43–88%across promptsGPT-5.6 Luna45–84%across promptsDeepSeek V4 Flash 073149–76%across promptsQwen3 30B A3B Instruct 250759–63%across promptsSeed-2.0-Mini51–71%across promptsGemma-SEA-LION v4.5 E2B-IT47–73%across promptsLing-3.0-flash55–65%across promptsGLM 5.3 Flash51–69%across promptsMistral Small 347–55%across promptsSolar Pro 439–61%across promptsLlama3 8B CPT Sahabat-AI v1 Instruct35–59%across promptsLlama 3.1 8B Instruct24–41%across promptsHunyuan A13B Instruct22–39%across prompts
MiMo-V2.6-Flash76%
Indonesia
Indonesian slang comprehension
Explaining what Indonesian slang words mean inside a sentence.
60 items18 held back19 modelsLast tested
Score spread
GPT-5.6 Luna42/42 100.0%GPT-6 Astra42/42 100.0%Gemma 4 31B41/42 97.6%DeepSeek V4.1 Flash41/42 97.6%MiMo-V2.6-Flash40/42 95.2%Llama3 8B CPT Sahabat-AI v1 Instruct40/42 95.2%GLM 5.340/42 95.2%DeepSeek V4 Flash 073139/42 92.9%Mercury 2.539/42 92.9%Claude Haiku 4.538/42 90.5%GLM 5.3 Flash38/42 90.5%Ling-3.0-flash37/42 88.1%Seed-2.0-Mini37/42 88.1%Qwen3 30B A3B Instruct 250737/42 88.1%Solar Pro 437/42 88.1%Gemma-SEA-LION v4.5 E2B-IT33/42 78.6%Llama 3.1 8B Instruct32/42 76.2%Mistral Small 332/42 76.2%Hunyuan A13B Instruct29/42 69.0%
GPT-5.6 Luna42/42 100.0%
Jawa
Rare Javanese word meanings
Explaining 43 Javanese words absent from every open source.
43 items16 modelsLast tested
Score spread
Gemini 3.5 Flash40/43 93.0%GPT-5.6 Luna35/43 81.4%GPT-6 Astra33/43 76.7%Gemma 4 31B26/43 60.5%DeepSeek V4 Flash 073122/43 51.2%DeepSeek V4.1 Flash22/43 51.2%MiMo-V2.6-Flash15/43 34.9%GLM 5.314/43 32.6%Mercury 2.513/43 30.2%GLM 5.3 Flash11/43 25.6%Gemma-SEA-LION v4.5 E2B-IT9/43 20.9%Llama3 8B CPT Sahabat-AI v1 Instruct4/43 9.3%Solar Pro 44/43 9.3%Nemotron 3.5 Lightning3/43 7.0%Llama 3.1 8B Instruct1/43 2.3%Mistral Small 31/43 2.3%
Gemini 3.5 Flash40/43 93.0%
Jawa
Everyday Javanese word meanings
Explaining everyday Javanese word meanings verified by native speakers, part of it held back.
199 items64 held back18 modelsLast tested
Score spread
Gemini 3.5 Flash128/135 94.8%Gemma 4 31B116/135 85.9%GPT-5.6 Luna112/135 83.0%DeepSeek V4 Flash 0731107/135 79.3%MiMo-V2.6-Flash100/135 74.1%GLM 5.398/135 72.6%GLM 5.3 Flash97/135 71.9%Claude Haiku 4.586/135 63.7%Ling-3.0-flash69/135 51.1%Llama3 8B CPT Sahabat-AI v1 Instruct61/135 45.2%Gemma-SEA-LION v4.5 E2B-IT60/135 44.4%Solar Pro 452/135 38.5%Qwen3 30B A3B Instruct 250751/135 37.8%Seed-2.0-Mini29/135 21.5%Llama 3.1 8B Instruct26/135 19.3%Nemotron 3.5 Lightning25/135 18.5%Mistral Small 325/135 18.5%Hunyuan A13B Instruct21/135 15.6%
Gemini 3.5 Flash128/135 94.8%
Jawa
Javanese speech levels
Choosing the Javanese register that fits the person being addressed.
46 items13 held back20 modelsLast tested
Score spread
multiple choice
GPT-6 Astra26/27 96.3%DeepSeek V4.1 Flash26/27 96.3%Gemini 3.5 Flash26/27 96.3%Gemma 4 31B26/27 96.3%DeepSeek V4 Flash 073125/27 92.6%GPT-5.6 Luna23/27 85.2%GLM 5.3 Flash22/27 81.5%MiMo-V2.6-Flash22/27 81.5%Mercury 2.522/27 81.5%GLM 5.321/27 77.8%Claude Haiku 4.520/27 74.1%Ling-3.0-flash18/27 66.7%Gemma-SEA-LION v4.5 E2B-IT16/27 59.3%Solar Pro 416/27 59.3%Qwen3 30B A3B Instruct 250714/27 51.9%Seed-2.0-Mini13/27 48.1%Llama3 8B CPT Sahabat-AI v1 Instruct13/27 48.1%Hunyuan A13B Instruct12/27 44.4%Mistral Small 311/27 40.7%Llama 3.1 8B Instruct9/27 33.3%
GPT-6 Astra26/27 96.3%
open answer
GPT-6 Astra6/6 100.0%DeepSeek V4.1 Flash6/6 100.0%Gemini 3.5 Flash6/6 100.0%Gemma 4 31B6/6 100.0%GLM 5.3 Flash5/6 83.3%Gemma-SEA-LION v4.5 E2B-IT5/6 83.3%MiMo-V2.6-Flash5/6 83.3%Mercury 2.54/6 66.7%GLM 5.33/6 50.0%Qwen3 30B A3B Instruct 25072/6 33.3%Llama 3.1 8B Instruct1/6 16.7%Llama3 8B CPT Sahabat-AI v1 Instruct1/6 16.7%Seed-2.0-Mini1/6 16.7%Mistral Small 30/6 0.0%
GPT-6 Astra6/6 100.0%
The only regional language measured so far is Javanese. Others such as Sundanese, Madurese, Balinese, Batak, Buginese and Minangkabau are still being collected: the sets do not exist yet, so there are no figures. We publish numbers only for what has actually been tested.
Every figure here comes from the same questions for every model, and the key does not change mid-test.