Grading systems -- how do you grade?

I just saw a post by Kevin Werbach that links to a post on the grading system used by Liz Lawley -- she gives only three grades: A for "good work," C for "mediocre but acceptable work" and F if "they really hadn’t mastered the material."

That got me thinking about grading. I will tell you how I grade and would like to hear how you grade.

For final grades, I curve students relative to the top person in the class, so they are not competing against each other. I also put them in heterogeneous groups and give significant bonuses to groups with low-variance, encouraging them to help each other.

Part of their grade is based on assignments. Since I give a lot of small, focused assignments, I had to devise a grading system that did not take a lot of time. I grade assignments as satisfactory and on time, satisfactory, but late, or not satisfactory. They get full credit for assignments that are satisfactory and on time and half credit for those that are satisfactory, but late. If they are not satisfactory, I explain why and they have the opportunity to re-submit a satisfactory answer for half credit. This method makes grading relatively quick and encourages students to keep up.

But, I am still grading at the course level, which is inappropriate for many subjects. For example, to say that someone received a C in an introductory statistics course means they did not learn significant parts of the material -- perhaps understanding descriptive statistics, but not hypothesis testing or estimation.

Instead of one grade, I'd prefer a fine-grained system in which one could, for example, pass "measures of central tendency," then "measures of variability," then "basic probability," etc. In that case, "passing" an introduction to statistics would mean passing each of a series of ordered modules and understanding all of the concepts and skills presented in the course.

I've advocated and used modular teaching material for many years, but always within the confines of the standard grading paradigm -- assign a letter grade from A to F for an entire course. With today's technology, we could combine modular teaching material with pass/fail grading at the module level. The technology is the easy part. Breaking up the traditional transcript -- our current system of grading and certification -- would be tough.

But, enough blue sky dreaming -- I am curious to know how others grade. I've outlined my grading system and that of Liz Lawley -- how do you grade?

Low pass rates in the San Jose State/Udacity experiment, but is pass rate a good metric?

Udacity, the MOOC platform company, plans to experiment with on online MS in partnership with Georgia Tech and they have also tried offering a few courses for credit in partnership with San Jose State University (SJSU). The Georgia Tech experiment is just starting, but we have some preliminary results from SJSU.

I got a copy of a portion of a presentation on the SJSU-Udacity experiment. As you see below, they compare pass rates of the Udacity sections with the traditional classroom sessions. The Udacity results are disappointing, but I think "pass rate" is an outdated, pre-Internet metric for these courses. More on that later, but first, here is the presentation excerpt:

In Spring 2013, San Jose State University (SJSU) collaborated with Udacity - a for-profit online start-up - to offer basic Mathematics and Statistics classes. The Udacity leadership appeared in a news conference with SJSU President Mo Qayoumi and Governor Jerry Brown to tout this public-private partnership as a means of increasing both access and graduation rates at SJSU. This same Udacity leadership appeared with State Senate Pro-Tem Darryl Steinberg in the rollout of SB520 and in the presentation of SB520 to the Senate Higher Education committee where they described the collaboration as part of their vision for SB520.

As part of this "experiment" at SJSU, success rates comparing SJSU students in the online version of the three courses versus SJSU students enrolled in the traditional face-to-face (F2F) versions of the same courses were collected. In addition, there were some non-SJSU students also enrolled in the Udacity online courses. Below are the preliminary results of this experiment.

MATH 6L: Remedial/Developmental Math:
Udacity online version: 29% pass rate (14/49 passed; 2 withdrew)
Face-to Face version: 80% pass rate
Non-SJSU students in Udacity version: 12% pass rate (6/50 passed; 13 withdrew)

MATH 8: College Algebra:
Udacity online version: 44% C-pass rate (8/18 passed; 2 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 12% C-pass rate (8/67 passed; 20 withdrew; 17 = Unauthorized Withdrawal WU)

STAT 95: Intro to Statistics:
Udacity online version: 51% C-pass rate (19/37 passed; 1 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 47% C-pass rate (21/45 passed; 8 withdrew; WU = 9)

The class sizes were small (these were not MOOCs) and the pass rates disappointing, but I am sure they learned from the experience. They are currently offering five classes, and those might yield better results.

But, is pass rate a reasonable metric of success in the Internet era? In my opinion, the notion of "passing" with a C, whether face-face or online, is a flawed metric of success for many courses. As I have said many times before, it is like putting old wine in a new bottle -- using new technology to mimic the past.

If you got a C in an introduction to statistics, you did not understand a lot of what was taught. Maybe you understood descriptive statistics, but not hypothesis testing. Instead of one grade, I'd prefer a fine-grained grading system in which one could, for example, pass "measures of central tendency," then "measures of variability," then "basic probability," etc. In that case, "passing" an introduction to statistics would mean passing a series of ordered modules and understanding all of the concepts and skills presented in the course.

I've been advocating and using modular material for many years, but always within the confines of the standard grading paradigm -- assign a letter grade from A to F for an entire course. With today's technology, we could combine modular teaching material with pass/fail grading at the module level. The technology is the easy part. Breaking up the traditional transcript will be tough.

Can we grade and give feedback on college essays automatically?

I am skeptical, but a New York Times article says the folks at edX will be doing just that using "artificial intelligence." They also say students will be able to improve their essays using feedback from the system. EdX promises to open the grading platform to others -- presumably as a Web service.

EdX has hired Vik Paruchuri to work on the service, which he developed for an automated essay grading contest sponsored by the Hewlett Foundation. The contest results are reported in a paper by the contest organizers, which concludes that automated essay scores correlate well with those of human readers. (Before you settle for that, read this critique of those results by MIT professor Les C. Perelman).

The Times article does not contain a link to a service and I can't find one on the edX web site, so I will remain skeptical. Grammar checking? For sure. Meaningful feedback? Show me the API or URL.

Can we use automated test essay graders as writing tutors?

Massive online classes are a hot topic -- they may disrupt and democratize education. Large classes offer economies of scale in the cost of developing and delivering teaching material, but grading does not scale as well. Multiple choice questions and simple computer algorithms can be graded automatically, but grading essays and other forms of assessment is labor intensive.

Might we automate essay scoring?

Kaggle is a company that organizes contests where teams of data analysts -- data miners -- are given a training data set, which they use to develop a predictive algorithm. In a recent contest, teams were given 16,000 student essays that had been graded by humans. The essays were responses to questions on state standardized tests. The contestants used this data to develop scoring algorithms that were then used on another set of test essays.

Randall Stross, writing in the New York Times, says the winning predictive algorithms were "eerily accurate" compared to human graders.

This $100,000 contest was sponsored by the William and Flora Hewlett Foundation, and is part of a broader program on automated grading. They recently published an analysis of the efficacy of commercial essay grading programs and a Kaggle blog post says they will run a second contest for grading of short-answer questions this summer. Three additional automated grading studies are in development.

Whether they are graded by humans spending a couple of minutes each or computers, the short-essay questions on standardized tests are not terrific indicators of writing ability.

Giving students feedback on rough drafts as they write would be a better application of this sort of technology. Writing online is increasingly important, and I have developed teaching modules on several types of writing, including short documents like these essays. I urge students to use spelling and grammar checkers, but that is rather shallow. Could short-essay grading algorithms be turned on their heads to give students feedback on the quality of their drafts while they are writing?