Wednesday, April 18, 2012

Robot grading

The interwebs are in an uproar over the latest annual software competition for machine-scoring essays.

See, for example, Jonathan Rees's take on this, the column at Inside Higher Ed, or Marc Bousquet's column, which has a beautiful explanation of a lit review that I want to stencil on the hands of all students:
That literature review in many circumstances will be comprehensive rather than merely representative. It functions as a warrant of originality in both professional and funding decisions (“We spent $5-million to study changes in two proteins that no other cancer researcher has studied,” or “No one else has satisfactorily explained Melville’s obsession with whale genitalia”). It offers a kind of professional bona fides (“I know what I’m talking about”). It maps the contribution in relation to other scholars. It describes the kind of contribution being made by the author.

Typically actual academic writers attempt to partly resolve an active debate between others, or answer a question that hasn’t been asked yet, what I describe to my students as “addressing either a bright spot of conflict in the map of the discourse, or a blank spot that’s been under explored.”
Bousquet's larger point is that if we want meaningful writing, we need to give meaningful assignments that matter to the students and make them feel as though they're part of a larger discourse community (agreed!).

But the best point I saw was the essay (in the comments section at IHE) that the machine graded as a "6" (perfect score). It makes no sense, but its grammatical structures are correct, and it must have hit some algorithmic high points for vocabulary and use of proper nouns. Sample quotation: "Teaching assistants are paid an excessive amount of money. The average teaching assistant makes six times as much money as college presidents." (I can't wait for this to be picked up and asserted as fact by certain higher ed-bashing news networks, by the way.)

A few thoughts:

1. If drill-and-kill grammar software never taught students how to make subjects and verbs agree, will the canned responses that the essay software have any better luck in helping students to write better?

2. A central tenet of writing pedagogy over the past 20+ years has been that writing is not so much a way of writing down what you think as it is a way of finding out what you think, especially through multiple drafts. That's something that most writers would say is true, too. Will students have a chance to find out what they think if they can write nonsense sentences and get a 6?

3. Where's the grading software that is going to spend half an hour going over an essay with a student as you sit face to face at a desk or on Skype, helping the student to understand how to work through an argument, sharpen the analysis, or even write a clear topic sentence?

4. This is the scary part: Will students write to the software and make everything correct but not bother to think about what they're really saying, if an absence of thought and fact garners them as much credit as actual thinking and supported arguments?

The perfect educational circuit of the future: student signs up for a MOOC course, interacts with other students for points toward a badge, consumes podcast lectures, takes online quizzes, and writes a content-free research paper graded by essay software that gives it a perfect score.

No comments: