Watson flexed its considerable computational muscle, trouncing two all-time Jeopardy champions in last night's game and made a puzzling stumble with its answer in the final round. It was the second of a three-game series, featuring Ken Jennings and Brad Rutter competing against IBM's supercomputer, Watson—and each other.
The round started with Watson and Rutter tied for first place each with a score of $5,000. Jennings was not far behind with a score of $2,000.
In about nine minutes, the IBM machine had left both humans in the dust, achieving a score of over $21,000. It was able to "buzz" in before either Jennings or Rudder and answered almost 100% of the questions correctly.
The questions in the second round seemed to consist completely of "encyclopedic" trivia on subjects like architecture, history and musicdeviating slightly from the first match, which included nuanced questions that tripped Watson up at times.
Despite placing what many considered odd bets, (one Daily Double wager Watson made, for $6,435, was so bizarre and specific in number that it generated laughter from the audience), Watson amassed over $35,000 by the end of the game. Its dominance in buzzing in early and answering questions correctly made the crash-and-burn in the final category round surprising.
The category in Final Jeopardy was, "U.S. Cities" and the answer was "Its largest airport is named for a World War II hero; its second largest, for a World War II battle" which both Jennings and Rudder correctly answered as "Chicago". Watson answered "Toronto" setting off a firestorm of questions and suppositions in Twitter and other sites as to how the IBM machine could come up with an answer so out of left field.
It's worth to note, that all contestants have to answer the final question and that Watson's displayed confidence level in knowing the correct answer was only 11 percent, so in all likelihood, the computer would not have buzzed in, according to Stephen Baker, author of "Final Jeopardy." Still, it's a mystery how the machine came up with that answer to the final questions"
An IBM spokesperson provided some insight. Watson had "Toronto" as its first possible answer with a low 13 percent confidence. The next possible choice was "Chicago"--the right answer--at 11 percent confidence. The way in which Watson comes up with answers is multi-faceted. For example, the categories in Jeopardy do not always indicate what the answer's going to look like and that frequently, the answer is not what the category (or clue) is asking for. In fact, according to the spokesperson, "Watson treats the category as weak evidence." Additionally, Watson understands synonyms and equates Toronto, a North American city with American/U.S. There really is quite a complex level of processing going on as Watson sifts through related information: there are eight American cities named Toronto; the Toronto Blue Jays play in the American league etc.
The spokesperson also noted that the issue was one of semantics. Consider the final clue: "Its largest airport is named for a World War II hero; its second largest, for a World War II battle"" In regards to the second part of that sentence, we humans automatically associate "second largest" with "airport", but Watson does not make the same association.
It's clear, that even a question that sounds so easy to our ears, involves incredibly sophisticated and high-level programming and design for Watson to even, although in low confidence, have the right answer as one of its possible answers. On to round three!