Can we use automated test essay graders as writing tutors?

Massive online classes are a hot topic -- they may disrupt and democratize education. Large classes offer economies of scale in the cost of developing and delivering teaching material, but grading does not scale as well. Multiple choice questions and simple computer algorithms can be graded automatically, but grading essays and other forms of assessment is labor intensive.

Might we automate essay scoring?

Kaggle is a company that organizes contests where teams of data analysts -- data miners -- are given a training data set, which they use to develop a predictive algorithm. In a recent contest, teams were given 16,000 student essays that had been graded by humans. The essays were responses to questions on state standardized tests. The contestants used this data to develop scoring algorithms that were then used on another set of test essays.

Randall Stross, writing in the New York Times, says the winning predictive algorithms were "eerily accurate" compared to human graders.

This $100,000 contest was sponsored by the William and Flora Hewlett Foundation, and is part of a broader program on automated grading. They recently published an analysis of the efficacy of commercial essay grading programs and a Kaggle blog post says they will run a second contest for grading of short-answer questions this summer. Three additional automated grading studies are in development.

Whether they are graded by humans spending a couple of minutes each or computers, the short-essay questions on standardized tests are not terrific indicators of writing ability.

Giving students feedback on rough drafts as they write would be a better application of this sort of technology. Writing online is increasingly important, and I have developed teaching modules on several types of writing, including short documents like these essays. I urge students to use spelling and grammar checkers, but that is rather shallow. Could short-essay grading algorithms be turned on their heads to give students feedback on the quality of their drafts while they are writing?