Sunday, August 02, 2009

Crowdsource Grading

From Cathy Davidson's "How to Crowdsource Grading" at HASTAC via The Chronicle:
So, this year, when I teach "This Is Your Brain on the Internet," I'm trying out a new point system. Do all the work, you get an A. Don't need an A? Don't have time to do all the work? No problem. You can aim for and earn a B. There will be a chart. You do the assignment satisfactorily, you get the points. Add up the points, there's your grade. Clearcut. No guesswork. No second-guessing 'what the prof wants.' No gaming the system. Clearcut. Student is responsible.

And how to judge quality, you ask? Crowdsourcing. Since I already have structured my seminar (it worked brilliantly last year) so that two students lead us in every class, they can now also read all the class blogs (as they used to) and pass judgment on whether they are satisfactory. Thumbs up, thumbs down. If not, any student who wishes can revise. If you revise, you get the credit. End of story. Or, if you are too busy and want to skip it, no problem.

This sounds lovely, in theory. But, as usual, I have a few questions:
  1. Since this is "mastery grading" rather than "quality grading," wouldn't this be one of the cases where A = Adequate rather than excellent? Some professors don't have a problem with that, of course, but it makes me uncomfortable, since the professor is the one ultimately putting the A on the gradesheet.
  2. What about the retro soul who, having paid Duke U's high tuition, wants to know what an outstanding scholar like Cathy Davidson thinks rather than what his or her peers think? One comment that I used to get from time to time if I relied a lot on group work was "I'm paying to see what the experts think, not what my classmates think, about my work," and there's some justice in that position.
  3. As a corollary of the previous point (and this comes up in Jane Tompkins's A Life in School, too, where a similar method is described): do students ever get curious about what exactly the professor is doing to earn her salary? I don't think this is a question that ought to be posed, but I wonder whether students think about it anyway.
  4. So there are no petty jealousies, no cutthroat grad students, and no factions that might influence a student's willingness to make someone rewrite a post? I don't know grad students who would behave this way personally, of course, but there's a lot of trust involved with this system.



Annie Em said...

I need to go read Davidson's blog posting for more details, but I can see having students self-police (which is really what it is more than a real evaluation with feedback) for a small portion of their overall grade (let's say discussion postings only worth 20%), but I agree with you: part of my job is to provide feedback to students which is 99% comments and then the (yes dreaded) grade.

Anonymous said...

Maybe at Duke this would work -- super responsible students with good academic skills, where all their other courses were cutthroat, this would be an interesting contrast / relief and so on.

But the most I do in this regard is have students give presentations, upon which they then give a short quiz, of their design, which they grade.

Other times, I've group presentations which were graded by the student audience -- a grade which then was used as a percentage of the grade I gave.

I do find they like making and giving minor quizzes, contributing in a small way to exam question banks, and having a small voice in grading, but not MORE than that.

undine said...

Annie Em, I'd be all for this for part of the grade, but I've never been a fan of contract grading. I used to use a version of that for grading discussion posts in online classes, but for students in certain majors, that was just a license to do as little as possible. If I specified responses, they'd write 10 words; if I specified a greater word limit, they'd write wordy sentences saying the same thing (basically, nothing). If you don't leave room for your own judgment of quality, you can't complain when students use quantity instead of quality to earn an A.

Profacero, I've given students the ability to have an input into the grade for group projects, too, and it has worked well. But turning the whole thing over to them?

undine said...

P.S. Re "cutthroat": Alex Havalais left a comment at Davidson's site that talks about how his students formed factions and undercut each other in just this way:

michele said...

I agree with previous commenters that for discussions, group presentations and the like, input from the class on grading is useful. I've used class input on both types of assignments before. But it is still my job to evaluate the performance of the students, and relying entirely upon their grading means I'm not doing my job.

I'd also suggest that there's a reason it's called "crowd" sourcing. That's because it's really only effective when a crowd is doing the selecting/evaluating. A handful of students in a seminar, or even the number that might enroll in a larger lecture are just not enough to guarantee that the "crowd" will come to an appropriate consensus, let alone the problem of factions and revenge evaluation. Crowdsourcing in its original marketing form is designed to operate with a large number of inputs, more than you're going to see in a single classroom.

undine said...

Michele, that's an excellent point about the numbers needed for true crowdsourcing. In a classroom setting--and especially with undergrads, who have only recently emerged from the horrors of social groups in high school--I'd be very dubious about the ability of people to weigh in objectively if grades were at stake.