Blog Post

Why IBM's Watson is Dumber than a 5th Grader

Why IBM's Watson Is Dumber than a 5th Grader

POSTED BY February 18, 2011 on www.nowyouseeit.net, reblogged on www.hastac.org: (and with apologies for missing punctuation and apostrophes in the reblogging code-to-codeswitching trail . . . )

I know you saw IBMs impressive Watson robot/computer/cyborg wipe the floor with some of Jeopardys smartest winners this week.   There is no question that it is brilliant AI (Artificial Intelligence) programming and that the team of computer scientists, coders, linguists, and other knowledge-makers have done an incredible job pushing robotics forward.  But is Watson smarter than a 5th grader?   Short answer:  no.   Longer answer:  well, in some things he is lots smarter and in others, he doesnt stand a chance.

Heres why:   Jeopardy operates on far clearer linguistic rules than ordinary speech and ordinary conversation.   A 5th grader, even a smart one, doesnt have Watsons data base so cannot begin to know all those answers to all those clearly formulated, explicit questions.   However, Watson doesnt remotely have a 5th graders life-long data base of language formations, colloquialisms, neologisms, and grammatical errors that still compute (i.e. you can figure out what is meant despite the errorif you are a 5th grader, but probably not if you are Watson).   Watson hears questions by anticipating the sentence patterns of the Jeopardy questions and, searching among hundreds or even thousands of possible grammatical place holders, figures out what the subject and the relevant adjectives and even a few verbs are, pulls those out of the sentence, searches for complex integrated matches from its gigantic data bases, and then goes for the answer with the highest probability of being right.

Impressive.   But where a 5th grader is far smarter is in not needing to be pitched a fact-based question in a standardized formulation.  The fun of Jeopardy is its predictability in the question partand then the ability or inability of various contestants to fill in the blanks.   The questions on the program always come in categories and, in fact, there are websites for humans (such as this one: http://www.pisspoor.com/jep.html)  which give you advice for preparing for the show by understanding how the questions are formulated, what categories the answers will be in and so forth.  All of that is key to winning, and to programming Watson.

It's also key to item-response (multiple choice) testing.   Jeopardy isn't life -- and multiple-choice tests assess how much you know in multiple choice tests but not logic, creativity, problem solving, extrapolation, inference, or inductive reasoning.  And it is in those areas untapped by Jeopardy and unsullied by multiple choice testing that just about any 5th grader (whether academically measured as gifted or not) is already doing pretty well.  That 5th grader knows all the variations of language, including the mistakes, and is learningfrom Legos or Pokemon or Harry Potter or Barbie dollsall kinds of subtle and complex cultural and intellectual rules that allow her to make inferences that are not about plucking the right fact from a data base.   The 5th grader is also learning sociality, collaboration, cultural norms.  Ethics too, fall in the category of questions where Watson is dumber than a 5th grader.    Why dont we sneak some  ice creamor are you afraid of what your Mom will say? is not an easy question for anyoneand I would not trust Watson with giving my kid the right answer, at least one that would be compelling.   He may, in fact, get it right but the process isnt the ethically-driven consideration of desire and punishment, respect for authority and independence,  coercion and friendship, and on and on and on.   Being a 5th grader is very hard work and Watson is rarely up to the challenge.

When I was a 5th grader, I wanted to grow up to do one thing:  to create Watson.  Its a weird dream for a little girl, I know, but when others were playing Barbie, I loved writing equations that would translate language into propositions.   I was lucky enough to win a scholarship to math camp where this odd take on the world got a lot of kudos from the camp counselors.  I was a philosophy major in college because math and computer science departments hadnt yet claimed the area that, back then, was called quantitative logic and what now might be computational linguistics.  Its also called natural language, writing code not based on 0 and 1 but on figuring out the ways humans actually speak and trying to render that into a language from which machines could learn and relearn.   If you have a computer program that learns you are writing French in the midst of your English sometimes and stops trying to correct your mistakes, it is using a natural language programming of a simple sort.   Watsons natural language skills are extraordinary.

My college thesis was full of what we called chicken scratching.   Equations.  Hundreds, maybe thousands of them.  The one sentence that was its subject was I hurt my ankle.   Four words.  The relationship of the I and the my in that sentence takes  lot of programing.  So does the definition of ankle since its differentiation from other parts of the leg is not absolute.   Hurt?  Sheeeesh.   At some point, I flipped into semantics . . . then English . .  . then, voila, history of technology which blends all of them.   But I still pause whenever I hear I hurt my ankle . . . and I am flabbergasted by Watson.

But Watsons dexterity with I hurt my ankle doesnt matter to most 5th graders.   The dilemmas and interactivity of learning and language that we all face operates on a different level, by different codes, than Watsons.  We have much to learn from Watson as he learns from us.   And we have much, of course, to learn about the fabulously complex ways we negotiate the world.  Admire Watson.  And tonight, dear friends, give that 5th grader (now, past, or future) a hug . . .   In so many ways that matter most, that 5th grader is smarter than the most brilliant Watson in the world.

96

3 comments

This comic is a bit not safe for work, but it makes the point rather well that there is plenty about the world that Watson simply cannot understand: http://www.thehighdefinite.com/wp-content/uploads/2011/02/2011-02-15-culturally-biased.jpg

97

I think the more specific issue here is this: computers just aren't that good at handling qualitative information like cute kittens. The amount of work even to produce a machine like Watson, capable of the relatively simple task of playing Jeopardy, is tremendous. To go beyond that and produce a computer capable even of thinking on the level of the average fifth grader, to use your example, is orders of magnitude more difficult.

81