In mid-February, IBM's
To create Watson, IBM had to engineer analytics technology that could sift through the equivalent of one million books and find a single correct answer -- all in less than three seconds. What's more, the computer had to be able to understand subtleties in the English language, such as puns.
Now IBM, which is headquartered in Armonk, NY, is taking on an even bigger challenge: How to transform Watson's talents into analytic tools for businesses in a wide range of industries.
Leading the charge is Rod Smith, vice president of IBM Emerging Internet Technologies. Xconomy New York talked with Smith about Watson -- and how the computer might help companies solve problems that are much more challenging than the Daily Double.
Xconomy: In a nutshell, what is the technology behind Watson's ability to answer Jeopardy! questions in three seconds?
Rod Smith: There were two things we really leveraged in Watson. First we figured out how to get Watson to take questions apart and look at many interpretations.
Then, in parallel, he kicked off searches on lots of different search engines.
As information came back, it was weighted -- it had a probability of being correct. That probability was based not only on the information, but also on the source that Watson used. In training for Jeopardy!, some sources were weighted much higher in terms of their ability to produce good answers.
A few hundred potential answers came back. So from there, Watson went through a process of deep learning. He said, "Let me look at these really closely now, and really quickly, to find the one answer I'm willing to press a button on." He quickly figured out which one was closest to the right answer.
X: Can you give me an example of where this might be useful in the real world?
RS: Let's start in medical. If you think about how doctors do diagnoses, they have to sift through lots of new information coming in all the time. Watson could be a good assistant to a doctor. If you give him a few symptoms, he can come back to you with four or five candidates that might help you narrow your diagnosis down.
The doctor will get links to those candidates -- URLs -- so he can look at the latest journal article about a certain drug treatment or something similar that can help with the diagnosis.
In emergency rooms, his speed could be helpful.
X: How can Watson's technology help in newer areas of analytics, such as the monitoring of social media sites?
RS: You can train Watson on a set of information, and then turn that around and learn from it. So think about this whole area of social networking. Watson can sift through sites like Facebook or Twitter. We're not just talking about analyzing consumer sentiment to figure out who's likely to buy your product. Watson would also be able to find trends. He can look for patterns in comments on Facebook and Twitter, and then say, "You know, this one pattern keeps coming up. People are reporting a problem. You might want to think about changing your design of this product."
Fewer customers these days are going to the company for answers; they're going to their favorite community or social network and trying to find solutions. That changes the whole game on how we as vendors have to help our customers gather that information and react to it.
X: What was the biggest challenge IBM faced in getting Watson to play Jeopardy! like a real person?
RS: One very tough challenge was figuring out how to get him to bet correctly. Initially we thought it would be a really easy algorithm. But sometimes he bet $342. Ok. How did he come up with that very specific amount? It turned out to be a lot more complex than we thought. We had to get him to take into account many different factors, like what the categories were, what was left on the board, and so forth.
Throughout the development of Watson, we had to constantly remind ourselves not to generalize. We couldn't try to get one algorithm to solve everything. We needed to develop lots of different specialized algorithms to do that one specific thing very well and come back to you with the right answer.
We're not sure yet if the betting algorithm can be applied in the real world. We're thinking about how we can link health care and costs. But we need a lot of information to do that. It's different than Jeopardy!, where you have a finite amount of money on the board.
X: In the end, Watson came up with the right answers really fast. Is speed a necessary component of analytics?
RS: It's sexy, but as I talk to customers I'm always asking them how are they gathering information to gain insights, and what can we do from an analytics standpoint to help them? A lot of the folks I talk to tell me that a three-second answer isn't really necessary. They're happy with answers that come back in 10 minutes or a half an hour. I had one customer say, "It's taking me two years to do certain analyses. If Watson could come back with an answer in a week, I'd be really happy."
More from Xconomy.com:
- Lexalytics Digests Wikipedia, Sees Text Analysis Markets Broaden to Include Search, Travel, Law
- New Speech Recognition Engine Under the Hood at Vlingo; Startup Dumps IBM and Nuance for AT&T
- New CEO for Datawatch
Arlene Weintraub is the editor of Xconomy New York. She can be reached at [email protected] and followed on Twitter @awjourn.
