Tuesday, November 14, 2006

Secret confession time

I actually enjoy writing exams.

To me, writing exams is like putting together a puzzle. You have all the pieces there: all of the topics that you want to cover on the exam (the ones you've covered in class since the last exam, along with the ones you want to reinforce/retest from previous exams. In CS, it's hard to have an exam that is not at least partially cumulative). And you have the ways in which the pieces can fit together: the concepts within the topics that you want to test. Putting these together in such a way that (a) the exam assesses the students' learning in an appropriate way and (b) the exam is do-able within the allotted time is, to me, an interesting challenge.

My strategy for writing exams has evolved over the years, but I've finally settled on something that seems to work well for me. First, I put off writing the exam until the day before. This is probably not the most effective working strategy, but I find that I just cannot write an exam until the day before. However, I probably start thinking about the exam about a week before I start writing it: what have we covered? what would be a good way to test that particular concept? would a problem like X be too weird/too esoteric for this crop of students? So by the time I sit down to write it, I already have some good ideas for questions.

Second, I sit down and list all of the topics that we've covered since the last exam, along with any other topics I want to include on this exam. For each topic, I list two categories: "base knowledge" and "master knowledge". The "base knowledge" category lists the basic things I want students to know about a topic, the things that I think demonstrate the most basic understanding of a topic. The "master knowledge" category lists things that demonstrate what I think constitute "mastery" of a topic: things that show that students understand the topic and can reason about it in unfamiliar contexts. (Or, to put it bluntly, the level of understanding needed to earn an A for that particular topic.)

Third, I take each topic and the knowledge lists, along with the scattered ideas for questions that have been floating around in my head all week, and start to structure each question. The knowledge lists help shape each question, and also help me tailor the exam to the exam context. For instance, an in-class exam will focus more on base knowledge than on mastery (maybe 80%/20%, sometimes 70%/30% depending on the level of the class), while on a take-home exam the mix may be closer to 60%/40%. I also think about the context when structuring the exam: on a take-home, for instance, I'm more comfortable writing a problem that is instructive as well as assessive (is that a word?)---a question in which the students learn something even as they are demonstrating their understanding of a topic.

Fourth, I treat writing an exam like writing a paper: I write a first draft, print it out, then leave it alone for a couple of hours before going back to edit it. This helps me disconnect from the exam a bit and go back to it with a somewhat fresh perspective. Often while editing, I will actually do the problem, to make sure that my assessment of the difficulty level and time required matches up with the actual problem I've written. I will sometimes bug my colleagues to help me with the wording on questions, too.

This process is a bit time-consuming, but as a result I am rarely unhappy with an exam that I've written. The process forces me to be thoughtful about how I'm assessing the learning in my classes. It forces me to really think about *what* I want my students to know and *how* I want them to demonstrate that to me. And I think ultimately, it's made me a more thoughtful teacher as well: while planning out a new unit, I'll often think "what do I want my students to demonstrate to me at the end of this?" And I hope that ultimately, that makes me a more effective teacher as well.


Iris said...

Hi Jane,

I enjoyoed reading this post very much; and if u can excuse me I have a couple of questions: 1) do u repeat questions u have developed from previous exams? i.e. do u give your students an access to your past exams?
2) How/What resources do u consult when u build ur exams, beside the technique you have already mentioned in your post.

Finally, can you provide me with an example of your exams so I can imagine the (base/ master) part of your post.

Many Thanks

Anonymous said...

Hello Prof. Jane,

Being on the other side, as a student, I think, nothing beats taking a well written exam. It brings together the key concepts of the class and usually this causes the student to remember these concepts long after the class is over.

The hard part is, of course, writing a good exam. I am sure you've heard of one student's review of course that he/she had taken. "The course was very thorough. Whatever wasn't covered in class was covered in the final exam."


Jane said...

Hi Iris,

Great questions! I do occasionally repeat exam questions---sometimes verbatim, but most of the time I modify them a bit. I'm now at the point where I have a collection of questions I've developed for various topics, and I try to keep notes on what worked/what didn't work with each question. Other sources I use for questions include questions my colleagues have written for their course exams (works really well for courses we all teach, like the intro courses), and also end-of-the-chapter exercises from other textbooks (the ones publishers send me for review but that I don't end up using).

Here's an example of something I might ask. In our second intro course, we talk about trees. So I might ask them to construct a certain type of tree, like a binary search tree, out of a sequence of letters or numbers (base knowledge), then construct a different type of tree out of the same letters/numbers (base knowledge again), then ask them to explain which of the two trees is more suitable for this sequence of letters/numbers, and why (master knowledge).

Anon, that is definitely true---writing a good exam is hard, and is somewhat of an art. I always do appreciate it when students say that they liked an exam, or like a particular question on an exam. Sometimes, I will ask students (on the exam or afterwards) to tell me which question they liked the best and which they liked the least, and why. I get great answers from them on this!

Amy said...

This is really interesting. I had always been the one to answer the questions on an exam, so it's neat reading about how you created the questions.

Your students are very fortunate to have you as their teacher. Too many teachers nowadays teach to earn a paycheck only, and don't care whether their students actually benefit from the teaching. You are clearly a refreshing exception.