Home
Jeopardy is a game show in which the dude asks trivia questions. Some questions are in puns, and all questions are in natural language. Computers are now pretty damned powerful, so IBM decided to pit their computer, Watson, against the two best Jeopardy champions of all time.
"The historic "Jeopardy" matches between IBM's Watson supercomputer and two representatives of the human race are almost over. Watson is winning. The best and brightest wetware is on the ropes.
The secret to Watson's success is a collection of algorithms that can analyze natural language and guess whether the results yielded by those algorithms stand a good chance of being correct. This "self-knowledge" is important because Watson poses the question to all of the algorithms at once, then ranks the results. If confidence is high on any single result, the computer triggers a little servo that pushes the same button as the competitors from meatspace do.
IBM calls Watson "DeepQA," a reference to DeepBlue, the computer that defeated human chess champion Garry Kasparov and a descendant of an earlier chess machine that borrowed the name Deep Thought from "The Hitchhiker's Guide to the Galaxy." It was built deep in the machine rooms at IBM's Thomas Watson labs in Yorktown Heights, N.Y., a building from the "Mad Men" era designed by Eero Saarinen.
Behind Watson's curtain
Watson's brain showcases several IBM technologies. The hardware is jammed into 10 refrigerator-sized racks filled with Power7 server blades. To be exact, there are 90 Power750 servers filled with four processors each -- and each processor has 8 cores, for a total of 2,880 cores altogether.
The software is built on top of IBM's UIMA architecture, which IBM created and then open-sourced. UIMA stands for "unstructured information management application," and it offers a framework where modules can work independently on the problem at hand and then vote on the best answer. The Apache project hosts the current UIMA version, as well as a number of common modules, but not all of the modules used on the show.
Much of the power comes from IBM's carefully curated collection of data. Jennifer Chu-Carroll, one of the scientists who has worked on the project since it began over four years ago, says that Watson excels, predictably enough, when the answer is a detail stored in its database.
"Watson is very good at things we consider facts," Chu-Carroll explains. "The Beatles category is one we think of as 'filling in the blank.' They give you something and ask what's missing. If we have the source, that's fairly easy for the computer to do. Of course that requires understanding that type of question and actually having the source."
She points out that the entire corpus of information is built out of largely natural language sources, including licensed Beatles lyrics. Watson is completely self-contained and does not have access to the Internet during the game. "We do have things from Project Gutenberg," explains Chu-Carroll. "Things like Wikipedia. But primarily free text that is available to us: dictionaries, encyclopedias, newspaper articles, things that...cover 'Jeopardy' topics well."
Unlike Google and its Books project, IBM chose to obey licensing rules. "If we don't have a license, we don't have it," notes Chu-Carroll. Nor did IBM include any structure from CYC, a controversial artificial intelligence project designed to categorize much of human knowledge in a structured database. Most of the information used by Watson is plain text searched by the computer.
Beating the "Jeopardy" clock
Chu-Carroll says that search is among the most time-consuming operations for the algorithm. The team studied past "Jeopardy" games, timed Alex Trebek's reading speed, and estimated that Watson would have three seconds to work on a problem before buzzing in. Some observers have noticed that Watson is often unprepared if the clue is short and read quickly. This is because Watson allocates the same amount of time regardless of the length of the question, something Chu-Carroll says may be fixed in the future.
During this time, Watson's algorithms work to identify the most likely possible answers. It boils down the million or so book-length chunks of text into 100 or so likely answers, the top three of which were displayed for the entertainment of "Jeopardy" viewers.
The researchers analyzed the "Jeopardy" game and devoted an "entire team" to deciding what to wager on the Daily Double, the confidence level to reach before buzzing, and so on. Watson also tracks the amount of money it has and the amount its competitors have -- details that change the odds and change the strategy. It knows enough to avoid "pulling a Clavin," a reference to an episode of the 1980s television sitcom "Cheers," where postman Cliff Clavin made it to the final round of "Jeopardy" only to lose a commanding lead by wagering too much.
What's next for Watson's technology? IBM plans to commercialize the software by approaching industries with huge collections of knowledge that must be sorted and searched by employees. The focus on using plain, relatively unfiltered, natural-language texts means that the system can be reprogrammed for a new domain just by choosing a different corpus of knowledge. Call center representatives, for instance, may have Watson search through the vast databases of product manuals and other texts to help find the best possible answer.
IBM frequently mentions creating a doctor's assistant, a tool that will help suggest potential diagnoses to the doctor who will make a final decision and bear the brunt of any second guessing during a malpractice lawsuit. Humans, it seems, are still necessary for some things.
"
WATSON BEATS THE CHAMPS BY $20,000 DOLLARS!
FINAL JEOPARDY:
"David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several of things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. "What US city" wasn't in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn't find much evidence to connect either city's airport to World War II. (Chicago was a very close second on Watson's list of possible answers.) So this is just one of those situations that's a snap for a reasonably knowledgeable human but a true brain teaser for the machine. "
"The historic "Jeopardy" matches between IBM's Watson supercomputer and two representatives of the human race are almost over. Watson is winning. The best and brightest wetware is on the ropes.
The secret to Watson's success is a collection of algorithms that can analyze natural language and guess whether the results yielded by those algorithms stand a good chance of being correct. This "self-knowledge" is important because Watson poses the question to all of the algorithms at once, then ranks the results. If confidence is high on any single result, the computer triggers a little servo that pushes the same button as the competitors from meatspace do.
IBM calls Watson "DeepQA," a reference to DeepBlue, the computer that defeated human chess champion Garry Kasparov and a descendant of an earlier chess machine that borrowed the name Deep Thought from "The Hitchhiker's Guide to the Galaxy." It was built deep in the machine rooms at IBM's Thomas Watson labs in Yorktown Heights, N.Y., a building from the "Mad Men" era designed by Eero Saarinen.
Behind Watson's curtain
Watson's brain showcases several IBM technologies. The hardware is jammed into 10 refrigerator-sized racks filled with Power7 server blades. To be exact, there are 90 Power750 servers filled with four processors each -- and each processor has 8 cores, for a total of 2,880 cores altogether.
The software is built on top of IBM's UIMA architecture, which IBM created and then open-sourced. UIMA stands for "unstructured information management application," and it offers a framework where modules can work independently on the problem at hand and then vote on the best answer. The Apache project hosts the current UIMA version, as well as a number of common modules, but not all of the modules used on the show.
Much of the power comes from IBM's carefully curated collection of data. Jennifer Chu-Carroll, one of the scientists who has worked on the project since it began over four years ago, says that Watson excels, predictably enough, when the answer is a detail stored in its database.
"Watson is very good at things we consider facts," Chu-Carroll explains. "The Beatles category is one we think of as 'filling in the blank.' They give you something and ask what's missing. If we have the source, that's fairly easy for the computer to do. Of course that requires understanding that type of question and actually having the source."
She points out that the entire corpus of information is built out of largely natural language sources, including licensed Beatles lyrics. Watson is completely self-contained and does not have access to the Internet during the game. "We do have things from Project Gutenberg," explains Chu-Carroll. "Things like Wikipedia. But primarily free text that is available to us: dictionaries, encyclopedias, newspaper articles, things that...cover 'Jeopardy' topics well."
Unlike Google and its Books project, IBM chose to obey licensing rules. "If we don't have a license, we don't have it," notes Chu-Carroll. Nor did IBM include any structure from CYC, a controversial artificial intelligence project designed to categorize much of human knowledge in a structured database. Most of the information used by Watson is plain text searched by the computer.
Beating the "Jeopardy" clock
Chu-Carroll says that search is among the most time-consuming operations for the algorithm. The team studied past "Jeopardy" games, timed Alex Trebek's reading speed, and estimated that Watson would have three seconds to work on a problem before buzzing in. Some observers have noticed that Watson is often unprepared if the clue is short and read quickly. This is because Watson allocates the same amount of time regardless of the length of the question, something Chu-Carroll says may be fixed in the future.
During this time, Watson's algorithms work to identify the most likely possible answers. It boils down the million or so book-length chunks of text into 100 or so likely answers, the top three of which were displayed for the entertainment of "Jeopardy" viewers.
The researchers analyzed the "Jeopardy" game and devoted an "entire team" to deciding what to wager on the Daily Double, the confidence level to reach before buzzing, and so on. Watson also tracks the amount of money it has and the amount its competitors have -- details that change the odds and change the strategy. It knows enough to avoid "pulling a Clavin," a reference to an episode of the 1980s television sitcom "Cheers," where postman Cliff Clavin made it to the final round of "Jeopardy" only to lose a commanding lead by wagering too much.
What's next for Watson's technology? IBM plans to commercialize the software by approaching industries with huge collections of knowledge that must be sorted and searched by employees. The focus on using plain, relatively unfiltered, natural-language texts means that the system can be reprogrammed for a new domain just by choosing a different corpus of knowledge. Call center representatives, for instance, may have Watson search through the vast databases of product manuals and other texts to help find the best possible answer.
IBM frequently mentions creating a doctor's assistant, a tool that will help suggest potential diagnoses to the doctor who will make a final decision and bear the brunt of any second guessing during a malpractice lawsuit. Humans, it seems, are still necessary for some things.
"
WATSON BEATS THE CHAMPS BY $20,000 DOLLARS!
FINAL JEOPARDY:
"David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several of things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. "What US city" wasn't in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn't find much evidence to connect either city's airport to World War II. (Chicago was a very close second on Watson's list of possible answers.) So this is just one of those situations that's a snap for a reasonably knowledgeable human but a true brain teaser for the machine. "
