94.2% of Wikipedia corrections fail to become test fixtures

Wikipedia edit history looks like a gold mine. Every time somebody fixes a typo, a before-and-after pair gets recorded, free, at scale, with a timestamp and an editor attached.

We mined it. We got 744 candidate corrections. Then we checked them one by one, and the number collapsed.

From 744 down to 43

A 5.8% pass rate. Nine out of ten candidates were unusable.

This was not a mining failure. The problem sat in the underlying assumption: we treated "somebody changed A into B" as a correction signal. It is not.

Most were not spelling fixes at all, but style edits, one word swapped for another equally correct one. Others were cases where both forms are valid and the editor simply preferred one. Some introduced an error that was fixed again a few edits later, meaning a raw mine would have taught us the mistake rather than the fix.

The slipperiest category came last: proper nouns that legitimately carry two accepted spellings. Changing one is not a correction, it is a preference.

How we checked them

Each candidate was read together with its sentence context, and one question was answered: is the "after" form genuinely more correct than the "before" form, rather than merely different?

The bar was set high on purpose. Anything that passes becomes a test fixture, meaning it turns into a fixed reference that decides whether a future code change counts as an improvement or a regression. A wrong fixture is worse than no fixture, because it locks the mistake in and makes correct fixes look like regressions.

One point that's up for debate is that verification was handled by just one person. Ideally, we would have had two independent annotators rate the set and calculated the agreement between them. Since we didn't do that, the 5.8% figure comes with an uncertainty that we haven't measured.

Why 43 matters more than 744

Had we stopped at 744 and used it directly, our correction rules would carry hundreds of "fixes" that fix nothing. Some would actively damage text that was already correct.

The gap between 744 and 43 is work that cannot be skipped, and cannot be sped up by using a larger model.

This is the part that tends to drop out of conversations about automation. Mining data is cheap and fast. Deciding which version is right is expensive and slow, because it requires somebody who knows the language, reads the context, and makes a decision they can defend.

A language model can help filter the first layer. It cannot take over the final call, because when it is wrong nobody finds out until the fixture has already locked that error into the system.

What this does not show

The 5.8% figure is not "Wikipedia's error rate". Indonesian Wikipedia is not what is being assessed here. What is being assessed is how usable edit history is as a source of automatic correction data, and the answer is: usable, provided 94% of what you mine gets thrown away first.

The number also does not generalise to other corpora. Edit histories from forums, social media, or editorial workflows have different characteristics and would very likely produce a different ratio.

And because verification was done by one person with no comparison rater, we do not know how far the figure would move in someone else's hands. That is a limitation we are recording, not one we have removed.