Saturday, November 21, 2009

HASTAC: Cathy Davidson on grading (redux)

Just for fun this morning, I've been revisiting the discussion of Cathy Davidson's "crowdsourcing grading" post over at HASTAC.

Davidson and the commenters make good points, especially about an internet culture in which everyone feels empowered--nay, entitled--to pass judgment on any random piece of writing available on the web. We're all being judged constantly anyway, goes the argument, and students will be judged by peers and outsiders in the workplace, so why not in a grading an assessment situation? I liked the clarification that Davidson offered in the comments:
Advocating crowdsourcing, contract grading, written evaluation and other forms of assessment (including self- and group-assessment, which studies show is often far more rigorous than external assessment if the forms of the assessment are set up in the correct way) is not to say we don't want standards. Quite the opposite. It is to say that there are forms of knowledge and standards of excellence that certain systems do not test, so having complementary systems is good.
She then goes on to say that feedback (comments) and not grades should be the focus, and I'd agree.

Davidson offers a more complete version at in which she says that her students asked her to rethink grading in terms of this new paradigm and she concludes "They were right." I'm not sure whether she means that she decided that she needed to give grades, or that she needed to hand over the process of grading to students, but she ends with this: "In the workplace and in our communities, we have to learn more about how to make judgments, to offer feedback, and to take criticism from those who are not 'the boss of us.'"

Well, yes, we do need to learn more about this process, but I'd say that part of learning about the process is giving feedback about what constitutes good and bad feedback. Nuanced, intelligent responses = good. (And you will never, ever, see a more polite and adulatory comment thread than the one at HASTAC.) Twitter piranha-like ganging up on a speaker = Lord of the Flies. Are the student graders assessed on their grading abilities, and, if so, who makes that determination--Davidson or the other students?

I guess what I'm trying to work through is that somehow, somewhere, there's always going to be an Invisible Hand of the Professor that's responsible for correcting the market forces of commentary and assessment. In reading through this material, I'm trying to figure out specifically where and how that invisible hand touches the grading process.


michele said...

I like the idea of the invisible hand of the professor correcting the market forces of crowdsourcing, and I think your example of the linked tweckling incident gets at why we need it. In the extensive comments attached to that article, several commenters suggested that the abuse got so bad because it was anonymous.

Although I agree with those who point out that it was not anonymous and that many of those posting no doubt knew the permanent nature of the record, I would say that the defining feature of the tweckle is that there are no significant consequences for those who are tweckling.

Really, if you think about it, what consequences did the twecklers experience? None that I can see. Even though who outed themselves (or are clearly identifiable) only experienced mild public disapprobation for their actions.

I think the same problem occurs in peer evaluations, which is why we are sometimes reluctant to encourage them, and why we might need the invisible hand of the professor there to correct them. In most cases where students seem to abuse their power to evaluate others, it is because there are few consequences for their actions. There usually isn't a penalty for poor peer evaluation other than perhaps some social snubbing. They may be censured by the professor, but the larger consequences to their academic careers or even their standing in the larger community are minimal at best.

When I think of my own evaluation practices, I see several controls that keep my comments and grade assignment from being unfair. I am evaluated on the comprehensiveness of my syllabi and assignments, I use rubrics to ensure fairness in grading, I would be quickly develop a reputation within my department and institution if I graded unfairly, and I would probably have more grade challenges (which no one likes dealing with) if students figured out that my assessment wasn't fair. So there are several consequences if I'm not fair in my evaluation.

There are much fewer consequences for students if they unfairly grade their peers, particularly if they agree as a group to inflate grades or attack another student or group in order to inflate their own evaluations.

This is where I see the need for an invisible hand to correct these aberrations, and the reason why we as teachers might be reluctant to hand over some of the responsibility for evaluation to students. Without serious consequences for thoughtless evaluation, thoughtless or even malicious evaluation can flourish.

That's not to say I'm against peer evaluation - I do try to incorporate some into each class I teach - however I still need to control the process, not as much because I'm an expert in the material, but as a guardian of the educational aims of fairness and equally applied standards for every student.

cat said...

Have you seen that the HASTAC Scholars are conducting a whole forum on this? There is obviously a lot of interest since, in a week, they have had about 2500 unique (human not bot) visitors and 110 meaty, thoughtful, serious discussions. Join the conversation!

undine said...

michele, I really like what you said about holding students responsible for their evaluation processe (as we're held accountable). There's an accountability that doesn't happen if everyone's anonymous--ironic, I know, coming from someone with a pseudonymous blog. Inviting criticism without accountability is asking for trouble (as with the Twitter debacle).

cat, the discussion over at the forum is good, but I'm reluctant to be the dissenting voice over at HASTAC.

CarlD said...

I think the hand should be visible and touch the process throughout. So I have the students use my assessment rubric, coach them in how to be constructive and how to balance criticism with appreciation, and make sure each piece of work is reviewed by more than one peer (with each new peer reviewing not just the work but the previous reviews). I emphasize that their job is to help each other do better, that it's fun when everyone does well and also improves my mood when I'm grading to everyone's benefit, and that because there's no curve, there's no cost for being helpful. And I don't have them assign the final grade; I think there's something a little ugly about grading, and that's on me.

undine said...

Carl, that's exactly the way it should work. The incremental process of having students review previous comments as well as the essays is great.

I think the invisible hand should be visible, which is why some portfolio systems would make me nervous if I were a student. As instructors, we know that a student can improve a lot and still stay in a C or B range, but the student who isn't given such information is going to wonder why improvement doesn't necessarily equal an "A." Feedback is the most important thing, but grades are feedback, too, and if they're going to be assigned after 15 weeks in any case, it seems unnecessarily cruel to withhold all information about what grade a piece of writing would receive. Of course, if the course were pass/fail or ungraded, that would be another story.