Passed the Exam, Failed on Real Writing

From test sentences to real writing

Language correction tools are almost always tested on sentences built specifically to test them. Short sentences, one error each, clean context. Our engine passed there, and for a while we felt pretty good about it.

Then we pointed it at writing people had actually produced.

The first number that came back was 3,741. Not 3,741 errors, but 3,741 things the engine thought needed fixing. Once we checked them one by one, most turned out not to be errors at all: place names treated as misspellings, conversational particles glued onto the word in front of them, technical terms "corrected" into words nobody meant.

A correction tool that's wrong more often than right isn't merely useless. It's dangerous, because people tend to follow a suggestion that appears on screen, especially one that looks confident.

Six rounds of fixes, and the number fell 27%

3,741 Round 1 2,729 Round 6

Every drop has its own cause, and none of them involved raising the engine's confidence threshold. We deliberately avoided that route. Raising the threshold does lower the number quickly, but that only hides the problem instead of fixing it, since the correct flags are the first to disappear. If a number looks good simply because the engine stopped working, that isn't progress.

The most surprising part came from the biggest fix. When one rule stopped gluing a particle onto the preceding word, that particle token didn't vanish. It fell through to the next rule, which had no data for it, and guessed. The total dropped, while the kind of error simply moved elsewhere.

The fix that lowered the count the most turned out to move the problem, not remove it.

This article itself miscounted at one point

7 of 7

The first six figures were each short by one. The cause: the line-counting command we used counts newlines, not records, and our output files didn't end with a newline, so every count lost its last record.

The seventh figure's problem was bigger: it was taken from a file that keeps getting overwritten every time the audit runs, so its contents had already changed several times since this article first went out. We dropped that figure rather than correcting it, since there was no frozen version of the file left to point at.

So the real decline is 27 percent, not the 32 percent we briefly wrote in the initial version.

What can't be concluded from this

The 27 percent figure is a drop in the number of flags, not a gain in accuracy. The two correlate, but they aren't the same thing. An engine that stops flagging anything at all would score a 100 percent drop and be completely useless.

We also haven't compared it against other tools yet, and two days is a very short window. The pace of fixing during two intensive work days says nothing about the pace over the coming month. If the curve flattens out later, we'll write that up too.