How Gartner Got IBM Watson so Wrong

ByPrimary Researcher

Executive Summary

  • Gartner has a pattern of making wrong predictions, which they demonstrated again by predicting that IBM Watson would succeed.
  • How much are Gartner’s accuracy problems related to their income from vendors like IBM?

Introduction

Watson IBM is now broadly understood to be a failure. Yet, Gartner gave IBM Watson their seal of approval back in 2014. However, within a few years from this time, IBM’s Watson had been accused of repeatedly exaggerating what Watson could do, not inserting medical domain expertise, and promising a series of proof of concept that never matched what was promised. How did Gartner get Watson so wrong and go along with IBM’s marketing department?

Join us as we cover how IBM bought the rating that Gartner wanted and how IBM used this purchased approval in its marketing.

Our References

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: You are reading one of the only independent sources on Gartner. If you look at the information software vendors or consulting firms provide about Gartner, it is exclusively about using Gartner to help them sell software or consulting services. None of these sources care that Gartner is a faux research entity that makes up its findings and has massive financial conflicts. The IT industry is generally petrified of Gartner and only publishes complementary information about them. The article below is very different.

  • First, it is published by a research entity, not an unreliable software vendor or consulting firm that has no idea what research is. 
  • Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services as a vendor or consulting firm that shares their ranking in some Gartner report. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. 

Gartner on IBM Watson

There is an eerie similarity between comments from IBM and Gartner regarding IBM Watson in the following quotations.

Watson Offers A Real Benefit or a Brand?

We begin by providing a quotation from Reuters that sets the stage before the Gartner prediction captured in the same article.

“Ed Harbour, vice president of Implementation at IBM Watson believes Watson is still unique in its field. “Are there other companies out there that offered AI-based systems and machine learning? Yes, there are,” he said. “However…I believe very strongly Watson is ahead of the competition and we’ve got to continue to push [to make Watson better]. No, I don’t think it’s something that anybody can just do.”

But according to Perlich, data scientists who want to create similar platforms as Watson could possibly pull from various offerings from the likes of Microsoft Azure, , or Data Ninja. But what those products don’t offer is the Watson branding. “And everybody’s very happy to claim to work with Watson,” Perlich said. “So I think right now Watson is monetizing primarily on the brand perception.””

That does not seem real. It appears to be a differentiation based on brand awareness. That is, Watson is used on many occasions is used because of its brand.

This may have been true in 2014, but Watson no longer has a good brand. Its brand has been dramatically degraded because it has publicly failed to produce value. It is also widely accused of being oversold by IBM. Those with access to what occurred with Watson internally have told us the following:

“There was a lack of medical knowledge on IBM’s technology side. They expected the AI to train itself from non curated medical records. Not only the data sets were not parsed to skip errors from manual input. But they also were not informative enough to derive the kind of insight they hoped.”

Why isn’t there a differentiation if IBM has invested at least a $ billion into Watson? According to Reuters, in 2014, this was the level of investment.

Gartner’s Watson Prediction

“Jamie Popkin, managing vice president at research firm Gartner, said IBM’s technology significantly improved how information can be used and managed. “I think they’ve developed something that takes us to the next step where information management needs to go,” said Popkin.

IBM said it decided to establish the unit because of strong demand for cognitive computing.

“We have reached the inflection point where the interest is overwhelming and we recognized we need to move faster,” said Stephen Gold, vice president of Watson Business.

Watson will be deployed on Softlayer, the cloud computing infrastructure business IBM bought last year.

According to Gartner, by next year there will likely be a large and growing market for Watson-derived smart advisors and it said that Crédit Agricole predicted that these systems will account for more than 12 percent of IBM’s total revenue in 2018.”

Curiously, none of this came to pass, and Gartner once again failed on another prediction. And one wonders if being paid by IBM may have influenced this accuracy level.

Go to top

Conclusion

Gartner takes a large amount of money from IBM every year. This influences their predictions and makes them go to bat for IBM applications and services.