As most people in ed-tech circles have heard by now, a much-touted MOOC experiment has ended in embarrassment.
In January, Udacity, a for-profit founded by Google and Stanford employee Sebastian Thrun to create customized online college-level video-based courses, announced that it would partner with San Jose State University to offer online versions of three SJSU courses. This was a small pilot: The three courses, in remedial and entry-level math and statistics, would be open to just 100 students each, half already enrolled at SJSU and half coming from other nearby community colleges and charter schools, for college credit at a cost of $150. Students would be able to seek extra help online from live tutors as well as from teachers or professors at their home institutions.
Udacity and San Jose State had reason to expect some good outcomes from this approach. In the fall of 2012, a San Jose State University lecturer supplemented his introductory course on circuits with online material from the edx MOOC (Massive Open Online Course) “Circuits and Electronics,” based on an MIT course. The pass rate rose from 59 percent for the San Jose State original version to 91 percent for the demonstrably more difficult MIT version of the course. In general, studies of online and blended learning have found completion and satisfaction rates similar to or slightly better than face-to-face courses alone.
However, that was not the case here. These were the pass rates for the three courses:
- 29 percent passed the Udacity Remedial/Developmental Math course compared to 80 percent in the regular face-to-face version of the course. Only 12 percent of non-SJSU students in the Udacity version of the course passed.
- 44 percent of San Jose students passed College Algebra compared to a 74-percent pass rate in the face-to-face version. Only 12 percent of non-SJSU students passed.
- 51 percent passed the most difficult class, Statistics, compared to 74 percent for face to face.
The final research report won’t be released until next month, and it’s too soon to know if this is an apples-to-apples comparison. Most of the students recruited to take these courses were either high school students or had already failed the entrance exam for college math, or had even failed remedial math once already. The fact that the most basic course had the worst pass rate indicates preparation might have been an issue.
Nevertheless, in the face of these poor results, the SJSU/Udacity experiment has been put on “pause,” with plans to resume in the spring semester. “SJSU remains firmly committed to its partnership with Udacity,” the president and provost said in a statement.
It’s pretty easy to guess why most of the students didn’t pass these courses. Most fundamentally, many of them lacked basic access to computers or the Internet at home. At the Oakland Military Institute, a well-regarded charter school, according to the San Jose Mercury-News, the 45 high school students taking the Udacity course had to compete for screen time at the same computer lab shared by every one of the school’s 700 students. The school had to provide class time to work on their courses and an extra teacher to help with their technical issues.
Second, the course was basically being designed and taught at the same time, which is hardly best practice for any kind of course. This resulted in many errors and lapses in communication. Professors were still writing curricula when the courses began, and two of the three courses went up without any deadlines or specific assignments. According to reports, students remained somehow unaware of the availability of the live tutoring until they were already far behind.
“We communicated our expectations poorly,” conceded Thrun in an interview with the education newsletter Edsurge. “We had two deadline-free courses. Especially in these classes, students fell behind. That was a mistake.”
The third issue could be dubbed the “worst of both worlds” problem. Students in these courses didn’t have face-to-face support from qualified instructors, as they would in the classroom. They also didn’t really have the leisure to explore their curiosity or to work at their own pace, as they would in a truly open online learning environment, because the course was shoehorned into a traditional 12-week semester with deadlines, exams and grades—a factor that probably contributes to the generally poor completion rates of MOOCs as well.
In the same EdSurge interview, Thrun cites “strong data” from Khan Academy, another source of free instructional videos, of the importance of flexible pacing for math students in particular. He also mentions “Math My Way,” a self-paced math program at the Silicon Valley community college Foothill College, which has been able to double pass rates just by giving students more time. Knewton’s online Math Readiness Course in use at Arizona State University, Alabama and UNLV has achieved higher pass rates than the traditional courses in part through self-pacing. And the course redesign experiments of the National Center for Academic Transformation, going back to 1999, show higher pass rates in introductory math courses through blended learning models with tens of thousands of students, many of which also include self-pacing.
It’s hard to say why Udacity and SJSU didn’t draw on more of the knowledge from these previous experiments. It’s also hard to say why they launched with an unfinished course, when they had completed material they could have drawn on, as in the more successful blended-learning experiment with edX and SJSU. SJSU faculty are using the course results to call publicly for a suspension of the partnership, writing, “Dealing with tough economic times by handing off education to private vendors and using public funds to increase online offerings through these vendors will not serve California well in the long run. Politicians’ well-intentioned efforts to increase access for students ignore a proven solution that we know will increase access: investing resources in more class sections.”
While this trial certainly proved that MOOCs aren’t magic, nor will it be the end of experiments with online delivery or for-profit partnerships in public higher ed in California or elsewhere. Upcoming offerings like the Udacity-Georgia Tech $7,000 master’s degree in computer science, and Coursera’s partnerships with 10 state university systems to increase flexible paths to degrees, hopefully will learn from this pilot that, when they offer an online course, they’d better make sure there is a student at the other end with a laptop, an Internet connection and reasonable preparation and support to learn.
Anya Kamenetz’s Digital/Edu blog covers innovations in learning and teaching. This article appears in The Hechinger Report.