Online CS courses: What does it mean, “willing to put in the effort”?
January 26, 2012 at 7:55 am 22 comments
Sebastian Thrun, who taught the massive on-line AI class with Peter Norvig at Stanford, has left Stanford to join a startup to offer more online courses. Their first course will teach complete novices how to build their own search engine, in seven weeks.
Can you do that? Do we know how to take people from zero to Bing/Google in seven weeks? The phrase that David Evans uses to describe this process is, “anyone who is willing to put in the effort will be able.” I’ve heard phrases like that a lot about CS1’s, and I wonder what it really means. “That student failed because he didn’t put in the effort.” I tend to believe that most CS1’s expect a huge amount of background knowledge, or expect a huge amount of reading and practice by students — that the teacher’s expectation of “reasonable” effort is not the same as the students. If Evans’ class is like the other online classes, with only 20% or so completing the course, maybe it’s aimed at students who probably could have taught themselves the content with a book, but weren’t motivated enough to do it — so only 20% could make the “effort,” because only they had enough prior knowledge to make the required effort “reasonable.”
How do you measure effort? I’m seriously wondering — what does it mean to put in “enough” effort? Are we measuring cognition, or time, or somehow “mental pain”? If you don’t have the prior knowledge, and have to go read lots of background literature, is that part of “enough” effort? Is effort measured in terms of time-on-task? If we don’t know how to measure “effort,” how do we know if our class is demanding too much “effort”?
Evans’s “Build Your Own Search Engine” course, however, will be “targeted to students with no background” in computer science, the Virginia professor says. (Evans is taking a year from his tenured post at Virginia to serve as Know Labs’ vice president for education.)
“The goal is to have a course that anyone who is willing to put in the effort will be able to take,” says Evans.
via Stanford open course instructors spin off for-profit company | Inside Higher Ed.
Entry filed under: Uncategorized. Tags: distance education, educational psychology, educational technology.
22 Comments Add your own
Leave a comment
Trackback this post | Subscribe to the comments via RSS Feed
Computing Ed Research - Guzdial's Take 
1.
Alan Kay | January 26, 2012 at 8:14 am
Hi Mark
Back to “how much help does person X need to learn Y?” There is definitely a spread of pre-disposition, both genetically and experientially. At least at this level, we have something similar to the prospect of learning a musical instrument (and I think, many other kinds of things).
The exciting prospect — not yet realized I believe — for computer assisted learning is the combination of self-pacing that books (outside of a structured class) allow, and the prospects of much better feedback on individual progress than books can furnish.
Some people will need longer to learn Y, and will often need specific kinds of advice based on their X. Sometimes a self-paced learner is lucky enough to be able to get advice from an expert that they can intersect with off and on — this is easier with most instruments than with many other fields.
I don’t think there is a philosophical barrier that blocks computers from eventually being able to give good tailored advice after being able to assess what a particular learner has been doing. And I also think there are subjects — and especially parts of subjects — that are within range right now.
I believe it should be a national priority to actually make this happen. (It was labeled as such by President Obama in the first months of his tenure, and there has been a little movement towards this, but nothing so far that would match up to the meaning of a real “national priority”.)
Cheers,
Alan
2.
allenderl | January 26, 2012 at 9:21 am
Hi Mark & Alan,
I took the AI course and was doing fine in it, but there was nothing automated about it other then the delivery. The delivery system was smart enough, but there was nothing remotely like an intelligent tutor involved. The course was taught by online lecture and quiz, so students could take their time learning each subject.
The biggest barrier (as with most CS1 classes) was that the time required was huge, even for a single pass through the lectures and homework, the course took about 10-15 hours per week. At least for me it was nice to have it available when I wanted it, but eventually I had to get back to my real life.
Laura
3.
Alan Kay | January 26, 2012 at 3:07 pm
Hi Laura
Just curious … was there anything in the lectures that couldn’t have been done more clearly with media? (for example, most people can read 4-5 times faster than a person can talk — and quite a lot of prose writing is much more clear than oral speech patterns).
What if you had media with little embedded AI programs that you could run and (non-destructively) change and play with?
This would likely cut down on the amount of time needed in many different areas of the Stanford process.
Then it would be nice to get much better assessment and advice than one could get from quizzes …
Cheers,
Alan
4.
allenderl | January 26, 2012 at 3:47 pm
They actually did a really nice job of demonstrating concepts using a whiteboard in animation style. In that way, it was a simple improvement on in-class lectures, because you could watch (and re-watch) what they wrote and get a good sense of the logic or behavior described.
The course was highly mathematical so this was a reasonably appropriate medium.
i was in the class for about 4 weeks before I had to drop it due to other time commitments, but I found the use of the media quite reasonable (and i was doing well enough in the class). BUT from an educational technology perspective, only the scaling issues were of interest to me. Because homeworks were due at the same time for everyone, the servers crashed every week for several weeks in the hours before the homework was due. The crashes led them to extend the deadlines…
All of the exercises (homework, quizes, and tests) were challenging multiple choice questions. The experience was only interesting because they are great instructors, not because of anything in the way the media was used except for how the whiteboard was used to animate ideas.
For the person below who commented on online learning being learning alone, au contrair, in this case the learning was very social — students set up scads of FB groups, mailing lists and boards where one could go for help or just to groan about how much work they gave us. It was a social experience with 140,000 people (including my spouse).
Overall, if I had had more time, I would have definitely finished the course. But as an Ed-Tech person, I think there is a long way to go to realize the promise of the online course experiences.
5.
timjohnson | January 27, 2012 at 4:11 am
Hi Laura, I don’t know of any distance learning/online degree class that doesn’t expect at least 12 hours of student effort. Yes it is hard and as you and your spouse were doing it together time issues would be even worse. I would have been surprised if you, as the female, had not given up because of time pressures.
I don’t know where this idea of not having to work much outside lecture times has come from. We used to say we were reading for a degree because that’s what we did, we read – lots.
I think that’s what we (lecturers) mean by, “willing to put in the effort”. You have to be willing to let go of “life” outside of your subject – it’s called focus. In economics it’s called, “willingness to pay”.
Sorry but getting a first degree, Masters or PhD is hard work and it gets more demanding with the higher degrees.
6.
Mark Guzdial | January 27, 2012 at 11:19 am
Tim, how is “willingness to pay” measured in economics? Most of the responses in this list are talking about hours-per-week outside of lecture, and that is one way to measure “effort,” but it’s still a measure that presumes an average student. If a student doesn’t have a lot of prior background, they are having to read pre-course material (which takes hours-per-week outside of lecture) apart from learning the new content, which means that there is more “effort” being expended, quite apart from the expected effort used in designing the course.
7.
Bri Morrison | January 26, 2012 at 9:21 am
I think that Thrun and Evans would define “enough effort” as “whatever it takes to master the content.” As Alan commented, this is different for different people which is one of the problems that online students suffer from.
In a face-to-face class it’s easy to walk into class and ask the person next to you if they had trouble with the homework or readings. Even if that person says “no” then perhaps another person around them chimes in and says “Yeah, I had trouble to until I realized blah.” This allows students to know that they are not alone in their confusion and perhaps prompts a question to the teacher.
In the online environment, practically everything (at least in computing, especially CS1) is done alone. And few students have the self-confidence to post in a discussion forum that they are having trouble mastering a topic.
I think in an online or self-paced book environment the key is to be able to help students over their confusion by helping them understand what they are confused about by educating them on misconceptions and to let them know where they stand in relation to others who have attempted the course / book / exercises (e.g., “Congratulations, you are one of 3 people out of 20 that got that exercise correct on the first try.” or “Don’t worry, the majority of people also missed that question. Try again.”
I don’t necessarily think the problem is not enough effort on the student’s part, but our expectation of what we expect them to learn and know without support scaffolding or a community.
8.
Cecily | January 26, 2012 at 10:23 am
I usually measure effort as time on task since that is one of the few factors that we can easily control that has been shown to correlate with learning. I can then measure time on task required for problems that I give and let students solve in class. As long as I use similar problems for homework, I can project approximately how long it will take them to do that. I shoot for about 1.5 hours of problem solving a week out of class(plus 1.5 hours of problem solving in class), plus any reading that they want to do, and some time for memorization.
As far as pain or scaffolding, I try to give my students lots of little problems at the beginning while they are learning- usually about two methods to write per class meeting. Typically, 60% of this programming is repetitive stuff they have done before. If less than 50% of it is repetitive, we do the problem in class together.
My students still have pain and tears, especially when we take our programming quizzes that are closed-note,closed-book, but they also get a chance to redeem themselves, and this seems to help a lot.
I think a more interesting question that this post begs is “how is teaching in a top R1 school similar or different to teaching for the masses or educating the bottom billion?” Certainly, there are a lot of smart kids that choose not to do an R1 school for whatever reason, but there are also a lot of people that probably require different pedagogy as you begin to reach a less selective population; my experience has been that usually less-selective populations need more opportunities to try.
9.
gasstationwithoutpumps | January 26, 2012 at 4:39 pm
So you are aiming a 4-hour/week total effort? Around here that is a 1-unit class in engineering—CS1 is usually a 12–15-hour/week class, with 3.5–5 contact hours and 8–10 hours working alone (or with other students).
10.
Cecily | January 27, 2012 at 11:13 am
I’m not sure how you ended up with 4– I am shooting for closer to 7 hours a week- 1.5 of problem solving in class, 1.5 of problem solving out of class, 1.5 of lectures/quizzes in class, and 1.5-ish of reading out of class and 1-ish of memorizing. My fast students will need less than 7 and my slow students will need more than 7, but hopefully nobody should need more than 10! I usually tell them a half a dozen things to memorize each week e.g. “import java.util.Scanner;” Also, we have a 3 course sequence of 3 credits each instead of the two course sequence with 4 credits. I suspect that the course you are describing is a 4 credit course? Or maybe 5 credits?? It is interesting to see how different schools set this up…
11.
gasstationwithoutpumps | January 27, 2012 at 12:44 pm
Yes, semester, quarter, and semester-in-a-quarter systems all result in different paces for courses.
We do semester-in-a-quarter, with 5-unit courses that in engineering take about 130–150 hours (35 in-class, 3 exam, and the rest as homework and lab time). Your ratio of in-class to out-of-class time is more like a humanities course here than an engineering course.
12.
Bonnie | January 31, 2012 at 7:54 am
I have always used the rule of thumb of 3 hours of work *outside* of class for every hour in class. So for a 3 credit course, one could expect to put in 9 hours outside of class. In reality, for computer science courses, that is a minimum – our courses don’t have a reputation for a heavy workload for nothing!
13.
Franklin Chen (@franklinchen) | January 26, 2012 at 2:25 pm
Even more important than the question of what “effort” is and how to measure it, there is the matter of quality of effort (as opposed to time). Two individuals could have the same motivation, same raw mental abilities, same amount of time spent, but one could have two orders of magnitude more effective study habits. This applies to any endeavor, but probably the most publicized has been in music, where “how to practice” is something many people do not know (especially “amateurs”) and therefore people put in huge amounts of effort with little gain. As an amateur musician who wishes to be efficient given very limited time, I try to be very careful of what kind of “deliberate” effort I apply. I wonder if there are CS pedagogical sites similar to music practice advice given at http://musiciansway.com/blog/tag/gerald-klickstein/ (which I have found tremendously useful).
14.
Rita Freudenberg | January 27, 2012 at 8:06 am
I realize that I regularly think the same when I look at my CS1 students, but this is because I already saw the effects.
I teach non CS majors and many of the students do not relate very well to computer science. I teach them during their first year at the university. Since several of them do not pass the exams at the first try, I got to see them again and I’ve seen some amazing effects. Students who had difficulties suddenly became counterparts to discuss with in the courses. For me, that shows that it was not the subject itself that posed the problem but their attitude. So now they are “willing to pur the effort” and it works. The willingness comes from insight in the necessity. And, as with everything you start to master, they start to like it.
15.
Kim Wilkens (@kimxtom) | January 30, 2012 at 2:01 pm
I signed up for the Stanford AI class as well and initially I found it extremely challenging yet rewarding, but after several weeks (6?), I just couldn’t keep up with the content in the time-frame they were spitting it out. As a full-time student and mom with some semblance of life, I just wasn’t willing to put in “that” effort. It makes me wonder why online classes need to go at the same pace as their brick-and-morter counterparts.
16.
The long tail may not hit a target: High school teachers « Computing Education Blog | January 31, 2012 at 10:00 am
[…] CS10K teachers? I’m not convinced. First, when I read the comments to posts about the the Stanford classes, or Fred Martin’s post, I’m struck by how many people took the courses who already knew […]
17.
Seth Chaiken | February 5, 2012 at 2:36 pm
How can we spread out the hours for college-level learning to make it more available to busy people? Media technology has the potential for doing this, but educators need to find out how.
It’s normal for a 1-semester 3-cr college course to take 3 contact hours plus 9 homework hours per week for about 13 weeks. I’m skeptical that objectives of such a course can be met without the student putting in the 150-160 hours of more-or-less undivided attention. But 12 hours every week is not feasible for most other than full-time, supported students.
So, perhaps a better problem is to find ways of spreading the same hours of study over a lot more than 13 weeks while maintaining effectiveness of learning. Effectiveness is lost when the course is too compressed, because of mental fatigue, missing opportunities for contemplation, and probably unconscious processing. But effectiveness is also lost when it’s spread out too thin. Is there any research about assessing the limits?
In the meantime, online instruction should be self-paced, and learners should be counseled on techniques such as keeping good notebooks and logs to track where they left off.
18.
Stanford partners with Coursera to offer more online courses: It’s what the faculty want « Computing Education Blog | March 13, 2012 at 4:47 pm
[…] how to build search engines. If you recall, they’re claiming that they’ll be able to teach complete novices. Audrey said that she was never asked what her prior background was. From the discussion forums, […]
19.
Teaching 4,500 students to program ubiquitous computing: Mike Richards and My Digital Life « Computing Education Blog | March 26, 2012 at 9:27 am
[…] on guaranteeing quality, in the sense that the assignments are do-able by students (see previous discussion on reasonable effort). What guarantees can you make about free courses? Does course quality matter? Share […]
20.
Udacity’s CS101: Who are you talking to? « Computing Education Blog | April 20, 2012 at 8:19 am
[…] (and maybe Coursera) model effective for more than reaching the best students, beyond the ones who are willing to put in the big effort? It’s an occupational hazard, of being a professor in a state university and of being a […]
21.
Collaborative Floundering trumps Scaffolding « Computing Education Blog | May 1, 2012 at 7:47 am
[…] to do with the floundering being successful. It seems to me that floundering is going to require greater cognitive effort, and thus, greater motivation/engagement to persevere. I also wonder about the complexity of the […]
22.
Sebastian Thrun bets education over driverless cars « Computing Education Blog | August 28, 2012 at 9:32 am
[…] Stanford. The low pass rates aren’t a problem, then. Rather, it’s using motivation and willingness to put in the effort as the filter, rather than wealthy and clout. They’ll still have few graduates, but it’s because […]