Only 899 of Our 3,289 New-Word Candidates Survived the Filter

Half the dictionary, zero appearances

We matched all 72,454 lemmas in our dictionary, one by one, against two million real Indonesian sentences.

38,559

Never appear 38,559Appear in corpus 33,895

It sounds like an accusation: more than half our dictionary goes unused. It isn't. A dictionary is supposed to hold rare words, and a list that only contains words that show up often in the news isn't a dictionary, it's a news summary.

The more interesting question is the reverse

What's more useful to know is the other direction: which words show up often across those two million sentences, yet aren't in our dictionary at all?

We filtered for words appearing at least ten times with no matching lemma. That produced 3,289 candidate words. The top ten are almost entirely tech vocabulary: test, website, link, live, smartphone, streaming. It looked like a ready-made list of loanwords for the dictionary.

That list is what then collapsed under its own weight once we looked closer.

899

First filter: how often a word starts with a capital letter. A word that's almost always capitalized is likely a brand or place name, not an ordinary word. Mall, for instance, is probably part of a building's name. More than a quarter of the candidates were cut here. Second filter: how often the word actually appears. It turns out only 30 words appear a thousand times or more; the rest average only one appearance per tens of thousands of sentences, too rare to call "commonly used."

What survived both filters is just 899 words, twenty-seven percent of the initial candidates.

The corpus also has a date

There's one more trap. The most frequent word on the entire list turned out to be test, followed by rapid in third place. Neither is tech vocabulary entering Indonesian. Half our corpus is 2020 news, and rapid is a fragment of "rapid test". It ranks high because of a pandemic, not a language shift.

This isn't a flaw in the corpus. A corpus is genuinely a snapshot of one period. The flaw is in how the result gets read: sorting by frequency then reading the top ten as "words entering Indonesian" is a conclusion the data itself doesn't support.

What can and can't be concluded

What survives: 899 words that behave like ordinary words, not proper names, and short enough to be reviewed by a human one by one.

What doesn't survive: the claim that 53.2 percent of any dictionary is "unused". A word's absence from a news-and-web corpus isn't proof nobody uses it. The corpus skews toward formal written language, and spoken language and regional varieties are barely represented in it.

We use the 38,559 figure above to decide which words need more careful handling in our correction engine, not to condemn which words deserve to be dropped. Words with no frequency data are precisely the ones we know least about, and therefore need the most care.

All the raw data can be downloaded and recalculated yourself. If something's wrong, that's exactly the news we're waiting for.