Can we use automated test essay graders as writing tutors?

Massive online classes are a hot topic -- they may disrupt and democratize education. Large classes offer economies of scale in the cost of developing and delivering teaching material, but grading does not scale as well. Multiple choice questions and simple computer algorithms can be graded automatically, but grading essays and other forms of assessment is labor intensive.

Might we automate essay scoring?

Kaggle is a company that organizes contests where teams of data analysts -- data miners -- are given a training data set, which they use to develop a predictive algorithm. In a recent contest, teams were given 16,000 student essays that had been graded by humans. The essays were responses to questions on state standardized tests. The contestants used this data to develop scoring algorithms that were then used on another set of test essays.

Randall Stross, writing in the New York Times, says the winning predictive algorithms were "eerily accurate" compared to human graders.

This $100,000 contest was sponsored by the William and Flora Hewlett Foundation, and is part of a broader program on automated grading. They recently published an analysis of the efficacy of commercial essay grading programs and a Kaggle blog post says they will run a second contest for grading of short-answer questions this summer. Three additional automated grading studies are in development.

Whether they are graded by humans spending a couple of minutes each or computers, the short-essay questions on standardized tests are not terrific indicators of writing ability.

Giving students feedback on rough drafts as they write would be a better application of this sort of technology. Writing online is increasingly important, and I have developed teaching modules on several types of writing, including short documents like these essays. I urge students to use spelling and grammar checkers, but that is rather shallow. Could short-essay grading algorithms be turned on their heads to give students feedback on the quality of their drafts while they are writing?

Wall Street Journal -- MBA writing is important and needs to improve

The Wall Street Journal just published an article on the decline in writing and presentation skills of MBAs. The article reports that GMAT (admission test) writing scores are down. They also interviewed employers at General Mills, Booz Allen Hamilton and Morgan Stanley, who are dissatisfied with the writing ability of newly hired MBAs. In response, business schools at The University of Pennsylvania, The University of Rochester, Northeastern University and Cornell University have increased the emphasis on writing in their MBA programs.

The article speculates that some of the decline may be due to increased numbers of foreign students and poor habits picked up on the Internet. My own belief is that writing and careful reading are more important in the Internet era than they were before. Internet writing is also different than writing for print -- it is more likely to be conversational, collaborative and concise.

I consider Internet reading and writing skills to be part of basic IT literacy, and have created several teaching modules on the topic. I am also working on a teaching module on presentations, which are also discussed in the article.

An example of collaborative writing at Educause 2010

We frequently write collaboratively on the Internet.  Writing a composite document, where each co-author is responsible for his or her own section, is relatively simple to organize, but finding a way for many people to collaborate on a more integrated document is difficult.

The organizers of last week's Educause educational technology conference were faced with a large collaborative writing challenge.  They wanted to create a white paper synthesizing the presentation highlights from the sessions concerned with research on teaching and learning. The problem they faced was that there were 23 50-minute sessions spread over three days. One person could not have attended, let alone documented every session.

Their solution was to seek collaborators among the three to four thousand attendees, many of whom would attend one or more of the 23 presentations.  They created a simple survey form (using Google Docs) and asked those who attended the sessions to answer four open-ended questions. The report will be written by an Educause staff member using the results of the survey.  We will follow up when that report is published.

Related links to:

• Posts on Internet writing
• Teaching modules on Internet writing

An ineffective email conversation

We cover the importance of precise, conversational writing on the Internet.

The following is the start of the transcript of an email conversation about ordering laptop computers that lasted more than nine months. The conversation eventually involved nine people. It was frustrating and ineffective because the people did not heed the basic guidelines we have suggested for Internet conversations -- they did not carefully read messages, did not respond to specific requests, did not meet commitments, and failed to quote previous messages to maintain context.

Many people wasted time in this exchange, and the funds were poorly spent. By the time the laptops arrived, the vendor had introduced a newer model.

The transcript begins as a conversation between me (LP) and the School administration (SA), School purchasing department (SP), and the University purchasing department (UP). It began in mid August 2009, when I informed the SA that my laptop warranty was about to expire, and I needed to replace it. The first reply was positive, but the commitment made in it was subsequently forgotten and ignored.

From: SA
To: LP
Date: Mon, 17 Aug 2009 23:26:44 -0700

I talked to the dean about getting a laptop for you using the grant money. The answers are as follows:
...You should get a new one no later than Oct. 1
...Your laptop should be no more than $2,200 as budgeted

Based on this, I sent a note to UP:

From: LP
To: UP, SA, SP
Date: Tuesday, August 18, 2009 2:21 PM

I would like to purchase a laptop combining funds from my foundation account and the $2,200 allocation from the Dean.
I would like to move ASAP because the warranty on my current laptop expires in a couple of days. Sorry for the rush.

And received this reply:

From: UP
To: LP, SA, SP
Date: Tue, 18 Aug 2009 14:29:52 -0700

1. Send the configuration from the Dell web site. [SP], can you forward a Req with the % that you are paying from the Stateside.
2. Also forward a PO from the Foundation

Then SP took responsibility from UP -- they said they would handle it.

From: SP
To: LP, UP, SA
Date: Wed, 19 Aug 2009 15:24:20 -0700

Thanks for the information. Also the grant is not processed via the Procurement Office (you are off the hook), this is a Foundation purchase. We have a Dell rep we work with on these types of purchases.

Also I have not gotten the information on the account being established. Once that is done, I will notify you and send you information on how to contract the rep so you can request an equote. I'd have you do that now but the equote has an expiration date.

I was worried about the warranty expiring, so asked if I could speed the process up by configuring the laptop sooner:

From: LP
To: SP, SA
Date: Wednesday, August 19, 2009 3:34 PM

> Once that is done, I will notify you and send you information
> on how to contract the rep so you can request an equote. I'd
> have you do that now but the equote has an expiration date.

Could I contact him or her now and start the process? I am in a rush because the warranty on my current laptop is going to expire in a few days, and I am *highly* dependent upon it.

I did not get a reply from SP, but did get a reply from SA:

From: SA
To: LP, SP
Date: Wed, 19 Aug 2009 19:13:07 -0700

Larry - let me look into the situation and I will handle

SA followed up with:

From: SA
To: LP, SP, UP
Date: Thu, 20 Aug 2009 17:57:08 -0700

On Tuesday I will check and see if there is some money in a grant I have that I could use to purchase a laptop for Larry.

A week later, I followed up with SA, quoting his earlier message:

From: LP
To: SA
Date: Thu, 27 Aug 2009 12:39:19 -0700

>On Tuesday I will check and see if there is some money in a
>grant I have that I could use to purchase a laptop for Larry.

I have a configuration ready to go -- are the funds available now? My current service contract has expired, and that is worrisome.

And received this reply:

From: SA
To: LP
Date: Thu, 27 Aug 2009 14:03:02 -0700

No the funds are not available - we have not yet received the grant approval - I expect to receive any day.

But, nothing happened, and my warranty had expired, so two weeks later, I followed up with SA, again quoting his earlier message:

From: LP
To: SA
Date: Thursday, September 10, 2009 4:16 PM

> No the funds are not available - we have not yet received the grant
> approval - I expect to receive any day.

Has there been further news on this?

And SA replied:

From: SA
To: LP, SP
Date: Fri, 11 Sep 2009 17:38:09 -0700

News on what?

This was just the beginning, eventually nine people were involved in the conversation, which stretched out over nine months. I got the laptop in June, 2010.

The Zen of Internet reading -- a bad example

Much Internet writing is conversational, requiring careful, mindful and often critical reading. This example illustrates hurried, thoughtless reading.

An enthusiastic, but rushed administrator at our university recently sent the following message to the university president and vice presidents, with a copy to our dean and faculty:

Subject: CSUDH Ranking...

Hello All… FYI on this news that CSUDH was ranked #9 by Forbes Magazine as a Best Buy in Online MBA Degree Programs.
You can read more at http://www.forbes.com/feeds/businesswire/2010/05/06/businesswire139265756.html

I followed the link to learn more, and it turned out that Forbes had not ranked us or any other school -- they merely posted a press release from a company called GetEducated.com on their news wire.

GetEducated maintains a database of online degree offerings and their cost. We were listed ninth cheapest among non-AACSB schools. This is indeed an accomplishment, but it is not a #9 ranking by Forbes Magazine.

The person sending the message had not read the Forbes post carefully.

After checking GetEducated, I sent a clarifying message to the same people who had received the initial email.

Subsequent to my message, they sent 19 emails congratulating us on our ranking by Forbes. The people sending those messages had quickly glanced at the initial email, felt good about it, and expressed their feeling. They did not follow the link in the initial message or carefully read my message.

This is an example of the problems that can arise when one is not a critical and mindful reader on the Internet. The problem is exacerbated when the reader uses a phone -- they will be less likely than desktop or laptop users to follow links and carefully read longer posts like the GetEducated press release.

Positive results using Twitter and a wiki in a collaborative writing assignment

Last semester, I used Twitter and a wiki to illustrate collaborative writing and the writing of short documents in a Network News assignment. The student response was positive, so I will repeat the experience.

I started a class Twitter stream for links to current events relevant to our class. I told the students to follow the feed, and posted about 125 items during the semester.

The writing assignment was near the end of the term. Each student selected a particularly interesting post, and summarized it and its relevance to the class in a short document. Once the summary documents were polished, the students added them to a wiki page, creating a collaboratively authored Network News report for the term.

For more detail, see this post on the assignment and survey results.

Mumbai terrorist attack: the 21-hour evolution of a co-authored document.

We discuss writing for the Internet, including co-authroing composite and co-authored documents and collaborative writing by small and very large groups.

The Mumbai terrorist attack provides an illustration of a co-authored document with many authors. The Wikipedia page on the attack was started at 18:20, 26 November 2008 by a user named Kensplanet. The initial posting consisted of two sentences:

The 26 November 2008 Mumbai terrorist attacks were a series of attacks by terrorists in Mumbai, India. 25 are injured and 2 killed.

Kensplanet revised his initial post 5 times during the next ten minutes, then others began to contribute. By 15:44, 27 November 2008, the article was 4,780 words long with many links. During this time, there had been 942 edits. Of those, 149 were anonymous (IP address only), 199 people made only one, and 93 authors could not be identified. Two contributors had made over 50 edits. Kensplanet, who started the article, had made 57.

The organization of the article also evolved during this period. While it began as two sentences, 21 hours later it was organized into 6 major sections with 5 sub-sections:

For the story of the evolution of another Wikipedia page, see this most interesting screencast by Jon Udell.

Should you write a paper or create an ad hoc blog?

I just gave a presentation on computer literacy courses at a conference. People typically write papers for publication in a conference proceedings, but I decided a companion blog would be more useful.

Creating an ad hoc blog for a project, event, or, in my case, a conference presentation, takes only a few minutes, but providing the content is somewhat more time consuming than writing a paper for publication.

My ad hoc blog ended up with 2,964 words in 9 posts. Writing it took longer than writing a paper of the same length with 9 sub-headings because of the inclusion of 76 links to references and enrichment material. Some of those 76 links would have been references in a traditional paper, but I would have left most of the enrichment material out. Since I was writing for a blog, and knew the reader could easily skip a link or follow it, I left them in. Adding the links is mechanical, but discovering and reading the enrichment material was time consuming. It also meant I learned more.

The division of the blog into 9 separate posts, each of which stands alone, allows the reader to focus only on portions of interest. Category tags and full-text search also help the reader focus and make sub-sections more discoverable.

Reading a blog is different than reading a paper. The author does not specify a reading sequence, and transitions between sections (posts) are not explicit. These drawbacks could be overcome by knitting the blog posts together into a linear paper or by preparing an overview post -- an extended abstract with links to the individual posts. Writing a traditional paper or an overview post would be relatively simple once the blog was complete, but it would still take time. (I will do both when I find time).

An electronic publication like an ad hoc blog is also mutable, whereas a conference proceedings is fixed once it is published. For example, I revised the Internet Writing post twice after publishing it. The blog also allows for comments and other feedback. Of course, an online conference proceedings can also be changed and feedback can be solicited.

The blog might also continue to grow. The first nine posts of the computer literacy blog were related to my presentation, but the tenth summarized a related presentation I heard at the conference. Some ad hoc blogs may continue after the event.

A final consideration is academic credit. Even if a blog is more effective than a paper, an academic writer may be reluctant to go that route due to publication pressure. The traditional notion of academic publication is out of date.

You often write term papers for classes. Would creating an ad hoc blog be preferable? Why or why not?