Tuesday, May 23, 2006

Teaching without tests: post-game analysis

A while back, I blogged about my experiment of nixing tests and giving all quizzes instead in my intro-level course. I have to say that the experiment was largely a success. I will definitely do this again; I'm happier with it and most importantly the students are much happier with it (well, most of them anyway).

Lots of things went well with the quiz model. First, I was able to assess where the class was in terms of learning and comprehending the material much more easily than in the past. I saw their progress every single week, and I could tell instantly what concepts were covered well and what were not. If I taught a concept poorly, I found that the students all made similar conceptual mistakes. If I taught a concept well, most students nailed it on the quiz.

Second, I found that I was able to be *more* creative with quiz questions than with exam questions. Because I had to fit in all the important concepts I wanted to test into a 10-15 minute timeframe, I had to think very carefully about what I wanted the students to demonstrate on the quiz and how I wanted them to show this. I wrote much better questions as a result.

Third, the grades dramatically improved, in that the quiz average is much higher than the test averages normally are. Students learned and retained the concepts much more thoroughly than in the past---I base this on the sophistication and thoughtfulness not just of the quiz answers, but of the questions the students raised in class too.

Finally, the students were much less stressed out about being evaluated. Students can really get worked up over CS tests, but somehow when the evaluation was presented as a quiz, there was no widespread panic. Perhaps because the stakes on a weekly basis are so much lower than they are with the 2-3 test model, at least perceptually.

There were very few negatives in this experiment. One hope I had was that I'd be able to identify struggling students earlier and get them the help they need earlier (tutoring, etc.). Well, identifying them is one thing, but getting them to actually come see you to discuss it is quite another. The struggling students, even once I identified them, avoided me like the plague, not unlike previous years. Not much I can do about that one, but still. Second, I had a nontrivial portion of the class who completely flaked out on the quizzes week after week--as in not showing up for them, even though they all knew about the "no makeups except in dire circumstances" policy. Why anyone would throw away 30+% of the final course grade is beyond me.

The one thing I feared the most--increased time commitment of making up the quizzes and grading them--never materialized. The workload was completely manageable, and there was only one time where I did not get the quizzes back in the next class period.

One final note: There was one (only one) particularly bad quiz, in which all of the students made the same conceptual mistake on one problem. They all got the syntax right, but the problem asked them to do something they hadn't done before, and rather than reason it out they all panicked. I could tell when the students handed in the quiz that they were completely demoralized and disappointed. I was actually able to turn this into a "teaching moment": emphasizing the importance of algorithm development when writing programs. Which is something I talk about all the time, of course, but the quiz experience really drove this home to the students in a much more effective way. A fortuitous moment!

I've now tried this in two different classes with great results. I can't say that I'll do this in every class, but I will be more willing to experiment with the quiz model in future classes.



Anonymous said...

Hello Prof. Jane,

I hope you're over the review blues. I really didn't know that Profs had to jump that many hoops especially in CS where I think they are in need of Profs. The more that I read of your blog the more I am glad that I didn't pursure academia.

One comment regarding quizzes is that it usually done better by students since the covered material is less. With large examinations or tests you can ask a question that requires applications from different questions.

Anyway keep fighting the good fight.


Anonymous said...

I'm continually amazed how much you genuinely seem to care for your students. Most of the academic blogs I read mainly consist of people complaining of every aspect of dealing with students -- grading, test-writing, office hours -- so it's a real treat to read someone who sees her students as more than just an annoyance!

Jonathan said...

Jane, I'm intrigued. One question...how big are your classes? I'm trying to think of whether I could make this work in my 200-student questions. The sheer difficulty of collecting that many pieces of paper without losing too much time might be an obstacle. Still, it would be cool if I could figure out a way to make it work.

apparently said...

Like Jonathan, I'm wondering if this will work for my 80 student mid-level class next semester. I've only taught it once before and it was a near disaster. You said the workload was not unbearable -- how can this be? Any tricks you can share?

Jane said...

SV, that's a good point; the students can probably focus better if there's less material covered. I hadn't thought about that, but it's no doubt true.

Anonymous, I am blessed for the most part with great students, which helps my job tremendously! Thanks for the kind words.

Jonathan and apparently, I'm not sure how this would work with large classes--the largest class I've tested this on is ~30 students. I probably spend just over an hour grading quizzes, typically. I would guess that my system would work "as is" for class sizes up to 50. After that, I'd say that you'd either need grading help (TAs) or you'd have to modify the questions a bit to be less open-ended. I'm not a huge fan of multiple-choice questions, but in a really large class that might be the way to go---or go electronic, if you use a course management system that has a quizzing feature and you don't care if students use books and notes on the quiz... Hope this helps a bit! (I'd be happy to answer questions more specifically...drop me an email.)

Wicked Teacher of the West said...

One of the best courses I ever took used weekly multiple-choice quizzes as the sole grading mechanism. (Biochem, not CS) The quizzes were incredibly challenging multiple choice questions. They really tested a higher level of thinking - it was imperative to understand the material. They were open note and open book, but required evaluating, analyzing, and synthesizing the material covered. It is very, very hard to write quizzes that test at that level, but if you can do it, grading is a snap.

I am thinking about including quizzes in my course. I've relied on all projects, but it takes me a long time to grade and return them, which means students don't get feedback quickly enough to be truly helpful. I appreciate your comments about the value to you in being able to review concepts they didn't understand and to the students in seeing how they're doing in a relatively low-anxiety way.

Anonymous said...

Everybody here seems to really like quizzes, but I have another point of view. Some years a go I took a chemistry class that had similar system with a 10 minute quiz on the beginning of the class once in a while and it was a disaster for me. I got so stressed out with the short time limitation that I could not think clearly and did not perform on the quizzes well - the grade I got from them did not really reflect what I learnt. Short quizzes actually cause me more anxiety than a regular test, because regular test lasts an hour or two, so I have time to calm down and think.

goodman said...

Without a base, an article cannot be written. This is why we have chosen search as the base for this beautiful article of ours.