Low pass rates in the San Jose State/Udacity experiment, but is pass rate a good metric?

Udacity, the MOOC platform company, plans to experiment with on online MS in partnership with Georgia Tech and they have also tried offering a few courses for credit in partnership with San Jose State University (SJSU). The Georgia Tech experiment is just starting, but we have some preliminary results from SJSU.

I got a copy of a portion of a presentation on the SJSU-Udacity experiment. As you see below, they compare pass rates of the Udacity sections with the traditional classroom sessions. The Udacity results are disappointing, but I think "pass rate" is an outdated, pre-Internet metric for these courses. More on that later, but first, here is the presentation excerpt:

In Spring 2013, San Jose State University (SJSU) collaborated with Udacity - a for-profit online start-up - to offer basic Mathematics and Statistics classes. The Udacity leadership appeared in a news conference with SJSU President Mo Qayoumi and Governor Jerry Brown to tout this public-private partnership as a means of increasing both access and graduation rates at SJSU. This same Udacity leadership appeared with State Senate Pro-Tem Darryl Steinberg in the rollout of SB520 and in the presentation of SB520 to the Senate Higher Education committee where they described the collaboration as part of their vision for SB520.

As part of this "experiment" at SJSU, success rates comparing SJSU students in the online version of the three courses versus SJSU students enrolled in the traditional face-to-face (F2F) versions of the same courses were collected. In addition, there were some non-SJSU students also enrolled in the Udacity online courses. Below are the preliminary results of this experiment.

MATH 6L: Remedial/Developmental Math:
Udacity online version: 29% pass rate (14/49 passed; 2 withdrew)
Face-to Face version: 80% pass rate
Non-SJSU students in Udacity version: 12% pass rate (6/50 passed; 13 withdrew)

MATH 8: College Algebra:
Udacity online version: 44% C-pass rate (8/18 passed; 2 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 12% C-pass rate (8/67 passed; 20 withdrew; 17 = Unauthorized Withdrawal WU)

STAT 95: Intro to Statistics:
Udacity online version: 51% C-pass rate (19/37 passed; 1 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 47% C-pass rate (21/45 passed; 8 withdrew; WU = 9)

The class sizes were small (these were not MOOCs) and the pass rates disappointing, but I am sure they learned from the experience. They are currently offering five classes, and those might yield better results.

But, is pass rate a reasonable metric of success in the Internet era? In my opinion, the notion of "passing" with a C, whether face-face or online, is a flawed metric of success for many courses. As I have said many times before, it is like putting old wine in a new bottle -- using new technology to mimic the past.

If you got a C in an introduction to statistics, you did not understand a lot of what was taught. Maybe you understood descriptive statistics, but not hypothesis testing. Instead of one grade, I'd prefer a fine-grained grading system in which one could, for example, pass "measures of central tendency," then "measures of variability," then "basic probability," etc. In that case, "passing" an introduction to statistics would mean passing a series of ordered modules and understanding all of the concepts and skills presented in the course.

I've been advocating and using modular material for many years, but always within the confines of the standard grading paradigm -- assign a letter grade from A to F for an entire course. With today's technology, we could combine modular teaching material with pass/fail grading at the module level. The technology is the easy part. Breaking up the traditional transcript will be tough.

Innovation from MOOCs and modular courseware

I gave a talk on innovation from MOOCs and modular courseware at EduSoCal 13 last Tuesday.

The presentation format is annotated slides, most of which are self contained -- images with long captions. A couple of examples:

• Money is flooding into educational technology
• Google is in a good position with their forthcoming play store for education, open source MOOC platform, Hangouts, Google Plus and Google Docs.
• Thirteen sources of modular courseware
• Fifteen historical new medium innovations
• Eleven areas for MOOC innovation

You can download the presentation here.

Google Play Store for Education and the new 3Rs

Google announced Google Play for Education at Google I/O yesterday.

The store will be geared toward K-12 schools and organized by subject and grade, as shown here.

The apps will be focused on the Common Core State Standards which spell out the language and math skills and concepts students are expected to learn for success in college and careers.

The subject-grade organziation and focus on The Common Core, will make discovery of good apps relatively efficient.

The store will not be online till next fall, but they are encouraging developers to build apps now.  I'd be surprised if folks like The Khan Academy and MathisPower4U were not far down this path already. If they and independent developers and creative teachers begin building apps, we may see a loosening of the textbook publisher's grip. (Apple can't be too happy about this announcement either).

This is a K-12 Play store, but it sounds like it could be used as remedial source for today's university students -- you would be astounded by the math and language skills of many of today's undergraduates.

Some may chafe at the standardization and focus of Google Play for Education, but it only addresses a portion of the school day and the material is essential -- the 3Rs of our time.

New wine for new bottles -- innovation from modular teaching material and MOOCs

I received a nice award and gave a talk on "Innovation from modular teaching material and MOOCs" at the Sloan Consortium 6th Annual International Symposium on Emerging Technology for Online Learning (#et4online).

The slides from the talk are now online here.

Each slide has text notes so they should stand on their own.  The presentation also contains quite a few links as well as a recording of Wolfman Jack howling.


If you wish, you can watch a video of the presentation, but that would probably take longer than reading through the PowerPoints.