Today's Chronicle of Higher Education includes a very interesting article on a new program by a former MIT prof that generates syntactically complex and grammatically correct yet nonsensical "babel" in order to test automated machine-grading tools. The babel-filled but gramatically correct essay gets read by a popular machine-grading device and scores 5.4 out of 6 . . . http://chronicle.com/article/Writing-Instructor-Skeptical/146211/
What do we make of this Fight Fire With Fire fable? Is it like MOOCs? Decry the beast, rage against the machine, and then go back to business as usual? Or is there something interesting to learn here?
HASTAC readers will guess the "interesting learning" bit is what I"ll suggestion: In fact, I hope people use this study and program well and don't just do a dopey, "See, humans do it better." Yes. And no.
If the machine-generated test essay were a syntactically clumsy and grammatically incorrect bit of babble, research suggests machine readers might well offer better and more thorough advice on writing better-constructed sentences than would a human in stressed circumstances--say, a teacher with 80 papers to grade in 24 hours.
Why is this important? Because other research shows that, if you are a teacher with 80 students, and you are responsible but overburdened and underpaid, you might be tempted to (a) do a quick and dirty job of offering feedback and that helps no one or (b) resort to a multiple choice format instead.
This auto-generator reminds us tools are made to be used well by humans . . . not to replace us. And the reverse is also often true. Instead of raging against the machine why not use it to help us see what we, as humans, do extraordinarly well and what machines do extraordinarily well, what we do poorly and what machines do poorly. The point of all the recent behavioral economics research (thanks again, Dan Ariely!) is that we think we are better at lots of things than we actually are. Hey! It turns out machines can be "overconfident" too. And we certainly know our faith in machines can be overblown--as well as kneejerk skeptical.
As usual, the machinic truth lies somewhere in between rage and romance . . . Technorealism, I call it.