San Jose State and Udacity release results for summer pilot courses

San Jose State University (SJSU) and Udacity have run two rounds of trial online courses. The pass rates in the first round were poor, but a press release from SJSU and a blog post from Udacity report improvement in the second trial, which was run this summer.

As you see, the pass rates were better in the summer than the spring for every class and the online students in two of the summer classes had higher pass rates than on-campus students.

Both Udacity and SJSU say they are "encouraged" by these results, but there were differences between the terms that confound the data and make it difficult to explain, say, the improvement in elementary statistics or the poor results in entry level math. The three courses that had been offered in the spring were revised, but the numbers of students in the classes and their backgrounds and motivations were also substantially different.

For example, the spring classes were relatively small and the students were all trying to earn credit. In the summer, there were more students and their motivations varied, as shown here:

This is not surprising. I have noted earlier that college credit motivates relatively few MOOC students and, in fact, non-credit students may turn out to be the most important audience for and customers of online classes.

The results of this trial are far from definitive, but this is a time of rapid innovation and such experiments are to be encouraged.

Speaking of innovation, I cannot bring myself to conclude this note without pointing out that "passing with a grade of C" seems like a poor metric for "success" in a course. That is the traditional university criteria, but the Internet gives us the ability to break courses into fine-grained modules and to provide more precise, informative measures of success.
-----

Update 9/12/2013

San Jose State has released a report on the Udacity experiment entitled "Preliminary Summary SJSU+ Augmented Online Learning Environment." The report author is not the preliminary investigator for the study, Elaine D. Collins, but was written "for" her by consultants at the RP Group.

It is reviewed in two posts:

1. Chronicle of Higher Education, "Few Surprises in NSF Report on San Jose State U. Test of Udacity Courses article"

2. Inside Higher Ed, "After weeks of delays, San Jose State U. releases research report on online courses"
_____

Update 11/18/2013

The academic senate at San Jose State is considering a resolution to restrict the power of the administration to unilaterally enter into course contracts -- to keep control with the academic departments.

The Academic Senate is expected to vote on Monday on a proposed policy that would forbid the university to sign contracts with outside technology providers without the approval of tenured and tenure-track faculty members in whatever department would be affected.

-----

Update 11/25/2013

The Chronicle of Higher Education interviewed two leaders on opposite sides of the MOOC debate at San Jose State and they discover common ground. Anti-MOOC professor Peter Hadreas learned that pro-MOOC professor Khosrow Ghadiri has been using MOOC material to run a flipped class -- not merely having students enroll in a MOOC -- and that it is very labor intensive, taking 80 hours per week. They agree that pedagogical innovation and experiments are good things.

I also noted that one of the professors is an engineer and one is a philosopher. Can you guess which is which? Which do you think spends 80 hours per week on his teaching?

-----

Update 12/13/2013

Georgia Tech designs its Udacity pilot to avoid failure. They distance the themselves from Udacity's debacle at San Jose State University.
-----

Update 12/18/2013

San Jose State will resume their testing of the material developed for three online courses -- Elementary Statistics, Introduction to Programming and General Psychology in the spring semester. Enrollment will be capped at 70 students for the statistics class, 150 students for the programming course and 35 students for the general psychology class. At least half of the seats for programming and statistics will go to SJSU students and the rest will go to CSU students on other campuses. The course will use SJSU's LMS, Canvas. This sounds like a traditional online class using the material that was developed for the Udacity trial.
-----

Update 2/2/2014

In the fall of 2012, students in two traditional sections of an introductory electrical-engineering course earned passing grades at rates of 57 percent and 74 percent, respectively. In an experimental third section, which was “flipped” to incorporate the MIT videos, the pass rate was 95 percent, but they did not adhere strictly to the MIT material.

In the spring of 2013 they also ran three sections, one of which used edX content. In the traditional sections, students passed at rates of 79 percent and 82 percent. In the experimental section, the pass rate was 87 percent.

The spring experimental section followed the MIT curriculum than did the fall 2012 section -- they covered more material.

The professor, Khosrow Ghadiri (mentioned above), said the pass rates of the spring-2013 trial should not be compared with those of the fall-2012 trial because the students learned different material and took different examinations. In the fall, he used the MIT content to help teach his own syllabus. In the spring, he used the MIT professor’s content and learning objectives.

Low pass rates in the San Jose State/Udacity experiment, but is pass rate a good metric?

Udacity, the MOOC platform company, plans to experiment with on online MS in partnership with Georgia Tech and they have also tried offering a few courses for credit in partnership with San Jose State University (SJSU). The Georgia Tech experiment is just starting, but we have some preliminary results from SJSU.

I got a copy of a portion of a presentation on the SJSU-Udacity experiment. As you see below, they compare pass rates of the Udacity sections with the traditional classroom sessions. The Udacity results are disappointing, but I think "pass rate" is an outdated, pre-Internet metric for these courses. More on that later, but first, here is the presentation excerpt:

In Spring 2013, San Jose State University (SJSU) collaborated with Udacity - a for-profit online start-up - to offer basic Mathematics and Statistics classes. The Udacity leadership appeared in a news conference with SJSU President Mo Qayoumi and Governor Jerry Brown to tout this public-private partnership as a means of increasing both access and graduation rates at SJSU. This same Udacity leadership appeared with State Senate Pro-Tem Darryl Steinberg in the rollout of SB520 and in the presentation of SB520 to the Senate Higher Education committee where they described the collaboration as part of their vision for SB520.

As part of this "experiment" at SJSU, success rates comparing SJSU students in the online version of the three courses versus SJSU students enrolled in the traditional face-to-face (F2F) versions of the same courses were collected. In addition, there were some non-SJSU students also enrolled in the Udacity online courses. Below are the preliminary results of this experiment.

MATH 6L: Remedial/Developmental Math:
Udacity online version: 29% pass rate (14/49 passed; 2 withdrew)
Face-to Face version: 80% pass rate
Non-SJSU students in Udacity version: 12% pass rate (6/50 passed; 13 withdrew)

MATH 8: College Algebra:
Udacity online version: 44% C-pass rate (8/18 passed; 2 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 12% C-pass rate (8/67 passed; 20 withdrew; 17 = Unauthorized Withdrawal WU)

STAT 95: Intro to Statistics:
Udacity online version: 51% C-pass rate (19/37 passed; 1 withdrew)
Face-to Face version: 74% C-pass rate
Non-SJSU students in Udacity version: 47% C-pass rate (21/45 passed; 8 withdrew; WU = 9)

The class sizes were small (these were not MOOCs) and the pass rates disappointing, but I am sure they learned from the experience. They are currently offering five classes, and those might yield better results.

But, is pass rate a reasonable metric of success in the Internet era? In my opinion, the notion of "passing" with a C, whether face-face or online, is a flawed metric of success for many courses. As I have said many times before, it is like putting old wine in a new bottle -- using new technology to mimic the past.

If you got a C in an introduction to statistics, you did not understand a lot of what was taught. Maybe you understood descriptive statistics, but not hypothesis testing. Instead of one grade, I'd prefer a fine-grained grading system in which one could, for example, pass "measures of central tendency," then "measures of variability," then "basic probability," etc. In that case, "passing" an introduction to statistics would mean passing a series of ordered modules and understanding all of the concepts and skills presented in the course.

I've been advocating and using modular material for many years, but always within the confines of the standard grading paradigm -- assign a letter grade from A to F for an entire course. With today's technology, we could combine modular teaching material with pass/fail grading at the module level. The technology is the easy part. Breaking up the traditional transcript will be tough.

Amherst and San Jose State University -- two approaches to MOOCs

There has been recent news and controversy surrounding the relationship of San Jose State University and MOOC providers Udacity and edX. I was at a conference the week before last and met Catheryn Cheal, Associate Vice President and Senior Academic Technology Officer at SJSU, who is administering the MOOC experiments at San Jose State.

She confirmed a positive result -- students taking a circuits and electronics class did better in a section that used edX material as a supplementary "text" than those in other sections. Based on that experience, they plan to offer more blended classes and will establish a Center for Excellence in Adaptive and Blended Learning.

That sounds reasonable to me. A professor had students subscribe to a MOOC in the same way as he or she would ask them to buy a book or read some articles. The results were encouraging, so the university established a center for research and training on that as a pedagogical technique.

But, SJSU has taken a second step that is less conventional. Ms. Cheal confirmed that they are now offering some classes -- elementary statistics for example -- that will substitute for on-campus classes. Students will get credit at San Jose State and the credits will be "transferable to most colleges and universities nationwide." (In a FAQ, they admonish the student to check with their school).

I am not anti-MOOC. I was at the conference I mentioned above to give a talk on the innovation in pedagogy, technology and school and social systems that I expect to come from MOOCs and modular courseware, but SJSU seems to be moving very fast.

The elementary statistics course mentioned above will be a general education course. On my campus, a faculty committee has to approve general education courses. Based on the course description, it seems to be a typical introduction to descriptive statistics and simple inference, but many majors offer such courses -- education, nursing, business, psychology, etc. Again, department and school curriculum committees traditionally specify curriculum.

I understand the desire to eliminate "bottlenecks" in student progress through the university and, as I mentioned, I expect far reaching innovation from MOOCs and modular courseware, but I am curious to know the decision to accept these particular courses for credit was made. (That is not to say it may not have been the correct decision).

We can contrast San Jose State with Amherst, which recently decided not to offer courses through edX, but to experiment on their own. That strikes me as a prudent middle ground. The university will learn and will become self-sufficient. For example, they might install an open source MOOC platform from Google or from Stanford/edX and let their faculty experiment by developing full courses or modules to supplement their current courses or they might decide to wait until hosted versions of those platforms become available. Amherst will learn and remain in control of its destiny.

Organizations often outsource peripheral functions in order to focus on their core mission. Outsourcing fast food service to Taco Bell might make sense, but teaching is one of the core missions of a university, and we should not rush to outsource it -- to a MOOC provider, a textbook publisher or anyone else.

-----

Update 5/12/2013

Google tells me the following universities are offering MOOCs using their open source platform, Course Builder: North Carolina State, Indiana, Cornell University, Universitat Politècnica de València, and the Indian Institute of Science, Bangalore.

They also say we can expect Course Builder to become increasingly easy for non-technical faculty to use, but will not comment on the possibility of offering it as a hosted service -- we will have to wait and see about Google MOOCs.

Update 5/13/2013

Robert McGuire provides more details on the blended circuits class mentioned above in this post.

Update 5/14/2013

San Jose State is not the first university to give credit for taking a MOOC. Six European universities gave credit to students who took Stanford's AI course online last year. Students had to go to the University of Freiburg in Germany to take a proctored final exam. San Jose State exams will be proctored using a web cam and screen capture by ProctorU.

Do you know of other cases in which college or university credit was awarded for successful completion of a MOOC?

MOOC monetization -- a free sample strategy

The high cost of education has been one of the drivers of interest in MOOCs, and nearly all MOOCs have been free up to this point. We are in the "invest to get users" stage.

I've heard speculation on ways to monetize MOOCs, and would like suggest one I've not encountered -- the free sample model.

We are starting to get some data on MOOCs. For example, Katy Jordan pulled together completion rate data on 26 of the MOOCs that have been offered by Coursera, EDx and Udacity:



MOOC critics point to these low completion rates as evidence that MOOCs are a Bad Idea, but those large enrollment numbers are still encouraging, so let's look further. Here is a plot of the number of views of the first and last videos each week in the Bioelectricity course at Duke University:



As you see, it is long-tail, y=1/x type graph. One way to look at the long tail is to conclude that the majority of enrollees were dissatisfied and the course was a failure. But an alternative explanation is that the folks were not really enrollees, they were browsers, and, like shoppers in any situation, most decide not to buy.

We can also think of their motivation. Maybe some were not interested in "completing" the course and getting "credit" -- they wanted something less complete, less formal. We have data on that from the Duke course as well. As shown here, fun and enjoyment was the strongest motivator.



Similar motivation was reported by students in a programming class. Sixty four percent of the students said they were interested in the course out of personal interest and curiosity, 33% for professional advancement and only 3 percent for university studies.

These three studies are small and limited, but perhaps we should be thinking about MOOCs as entertainment and lifelong learning as well as for formal education and vocational training, which brings us to free sample pricing. We can view enrollment over time as follows:



The spike in the first few weeks is mostly browsers -- people who want to see what the course is like and what the workload is. They would be getting a free sample.

After a few weeks, the enrollment decline slows and we are left with the students who are interested in completing the course. As we have seen, some are seeking formal credit and others entertainment and the joy of lifelong learning. Those people would presumably be willing to pay for the course. How much?

San Jose State University is currently running a trial in which students will get credit for Udacity courses for $150. The results of that pilot study are not yet in, but $150 is less than the University price.

How about the non-credit students? Let's say a typical MOOC runs 10 weeks and the first three weeks are free. What would people pay for seven weeks of entertainment or lifelong learning? Would it be as much as a movie? A hardback book? A steak dinner?

We can noodle around with a simple spreadsheet and get some napkin-sketch revenue estimates. For example, let's assume a for-credit price of $50 for a certificate of completion and a non-credit price of $20. The median number of students completing the MOOCs shown above was 2,777. Assuming that ten percent of those are for-credit students and the MOOC can be offered four times a year, we get annual revenue of $214,000. If we use the mean completion rate of 4,447, the revenue estimate increases to $343,000.

Note that under these assumptions, the bulk of the projected revenue is from the entertainment students, not for credit students.

If an organization offers several courses, are numbers like that sufficient to cover the overhead of management and administration, production, software refinement, community facilitation, hosting and bandwidth, etc. and pay a teacher who is full time on a single course?

A question for any good teacher reading this -- how effective a teacher would you be if you only taught one course and spent full time delivering and improving it?

-----

Update, 4/17/2013

Coursera revealed that they had revenue of $220,000 in the first quarter after starting to charge for verified completion certificates (http://bit.ly/12nd8Rb).  The income was from students paying between $30 and $100 per certificate.  Coursera co-founder Daphne Koller said prices were "around" $50.

This gives us a first cut estimate of what someone hoping to get job or school credit for a course might be willing to pay.  No doubt, the going price for certified completion will be higher than the price for un-certified completion for entertainment and curiosity.

(Coursera uers who pay for certification have to submit a photo ID of themselves to the company and are also tracked based on their “unique typing pattern” to ensure that people who take tests or turn in assignments are who they say they are).
-----

Update 5/23/2013

I posited a simple characterization of people signing up for MOOCs as either "browsing" or "enrolling." Phil Hill presents a more nuanced characterization in his blog post on blog student types, classifying students as no-shows, observers, drop-ins, passive participants and active participants. Details are presented in the paper called Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses.
-----

Update 11/27/2013

This post in the Chronicle of Higher Education argues that MOOCs have failed in higher education, and are better suited to vocational training and lifelong learning.
-----

Update 12/1/2013

Prominent MOOC provider Udacity has shifted emphasis from college credit courses toward vocational training and lifelong learning.
-----

Update 12/9/2013

A University of Pennsylvania study of a million enrollees in 16 MOOCs shows they have relatively few active users and low completion rates.

• Course completion rates are very low, averaging 4% across all courses and ranging from 2% to 14% depending on the course and measurement of completion.
• Across the 16 courses, completion rates are somewhat higher, on average, for courses with lower workloads for students and fewer homework assignments (about 6% versus 2.5%).
• Variations in completion rates based on other course characteristics (e.g., course length, availability of live chat) were not statistically significant.
• The total number of individuals accessing a course varied considerably across courses, ranging from more than 110,000 for “Introduction to Operations Management” to about 13,000 for “Rationing and Allocating Scarce Medical Resources.”
• Across all courses, about half of those who registered viewed at least one lecture within their selected course. The share of registrants viewing at least one lecture ranged from a low of 27% for “Rationing and Allocating Scarce Medical Resources” to a high of 68% for “Fundamentals of Pharmacology.” 

The California legislature may endorse credit for MOOCs.

Governor Brown is following in his father's footsteps.

Governor Pat Brown is remembered for developing the California master plan for higher education in 1960. His son, Jerry Brown, the current Governor of California, may be remembered (for better or worse) as an Internet-era reformer of higher education.

Last fall, Governor Brown signed a bill designed to generate open text books, in order to alleviate the rapid rise of textbook prices compared with consumer prices, shown below. (Click images to enlarge).

This spring, San Jose State University is running a trial offering of credit for massive online courses from Udacity, a private company. The hope is that this will reverse the rising cost of tuition and fees.

Now the California legislature is considering a bill requiring the state’s public colleges and universities to give credit for faculty-approved online courses taken by students unable to register for oversubscribed classes on campus.

For a glimpse of the thinking behind these measures, this is what the Governor had to say higher education in his January State of the State speech:

With respect to higher education, cost pressures are relentless and many students cannot get the classes they need. A half million fewer students this year enrolled in the community colleges than in 2008. Graduation in four years is the exception and transition from one segment to the other is difficult. The University of California, the Cal State system and the community colleges are all working on this. The key here is thoughtful change, working with the faculty and the college presidents. But tuition increases are not the answer. I will not let the students become the default financiers of our colleges and universities.

And, it's not just California. In the supplemental notes to President Obama's State of the Union speech, under the heading "Holding colleges accountable for cost, value and quality," we read a call for revision of the federal student aid system, allowing for new measures of value and new system of accreditation:

The president will call on Congress to consider value, affordability, and student outcomes in making determinations about which colleges and universities receive access to federal student aid, either by incorporating measures of value and affordability into the existing accreditation system; or by establishing a new, alternative system of accreditation that would provide pathways for higher-education models and colleges to receive federal student aid based on performance and results.

If you want to dig deeper, you can see the text of the ambitious bill introduced February 21 and some March 8 amendments.

Tuition and fees

Textbook prices

-----

Update 4/6/2013

The California State Senate unanimously passed Senate Bill 520, authorizing the granting of credit for courses taken online. The bill now goes to the Assembly.

-----

Update 8/2/2013

California puts SB 520 on ice. State Senator Darrell Steinberg, who introduced the bill, has put it on hold in the face of faculty opposition and to take time to evaluate new online programs proposed by the University of California, California State University and California Community College system. For example, the California State University will offer 35 online courses statewide.