Low pass rates in the San Jose State/Udacity experiment, but is pass rate a good metric?

Udacity, the MOOC platform company, plans to experiment with on online MS in partnership with Georgia Tech and they have also tried offering a few courses for credit in partnership with San Jose State University (SJSU). The Georgia Tech experiment is just starting, but we have some preliminary results from SJSU.

I got a copy of a portion of a presentation on the SJSU-Udacity experiment. As you see below, they compare pass rates of the Udacity sections with the traditional classroom sessions. The Udacity results are disappointing, but I think "pass rate" is an outdated, pre-Internet metric for these courses. More on that later, but first, here is the presentation excerpt:

In Spring 2013, San Jose State University (SJSU) collaborated with Udacity - a for-profit online start-up - to offer basic Mathematics and Statistics classes. The Udacity leadership appeared in a news conference with SJSU President Mo Qayoumi and Governor Jerry Brown to tout this public-private partnership as a means of increasing both access and graduation rates at SJSU. This same Udacity leadership appeared with State Senate Pro-Tem Darryl Steinberg in the rollout of SB520 and in the presentation of SB520 to the Senate Higher Education committee where they described the collaboration as part of their vision for SB520.

As part of this "experiment" at SJSU, success rates comparing SJSU students in the online version of the three courses versus SJSU students enrolled in the traditional face-to-face (F2F) versions of the same courses were collected. In addition, there were some non-SJSU students also enrolled in the Udacity online courses. Below are the preliminary results of this experiment.

MATH 6L: Remedial/Developmental Math:
Udacity online version: 29% pass rate (14/49 passed; 2 withdrew)
Face-to Face version: 80% pass rate
Non-SJSU students in Udacity version: 12% pass rate (6/50 passed; 13 withdrew)

MATH 8: College Algebra:
Udacity online version: 44% C-pass rate (8/18 passed; 2 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 12% C-pass rate (8/67 passed; 20 withdrew; 17 = Unauthorized Withdrawal WU)

STAT 95: Intro to Statistics:
Udacity online version: 51% C-pass rate (19/37 passed; 1 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 47% C-pass rate (21/45 passed; 8 withdrew; WU = 9)

The class sizes were small (these were not MOOCs) and the pass rates disappointing, but I am sure they learned from the experience. They are currently offering five classes, and those might yield better results.

But, is pass rate a reasonable metric of success in the Internet era? In my opinion, the notion of "passing" with a C, whether face-face or online, is a flawed metric of success for many courses. As I have said many times before, it is like putting old wine in a new bottle -- using new technology to mimic the past.

If you got a C in an introduction to statistics, you did not understand a lot of what was taught. Maybe you understood descriptive statistics, but not hypothesis testing. Instead of one grade, I'd prefer a fine-grained grading system in which one could, for example, pass "measures of central tendency," then "measures of variability," then "basic probability," etc. In that case, "passing" an introduction to statistics would mean passing a series of ordered modules and understanding all of the concepts and skills presented in the course.

I've been advocating and using modular material for many years, but always within the confines of the standard grading paradigm -- assign a letter grade from A to F for an entire course. With today's technology, we could combine modular teaching material with pass/fail grading at the module level. The technology is the easy part. Breaking up the traditional transcript will be tough.