Do Gartner's MQs Have Any Legitimacy? - Brightwork Research & Analysis

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

Vendors and consulting firms that have paid Gartner never tire of showcasing their scores in various MQs.
We answer the question of one person who asked whether the MQs have any legitimacy.

Introduction

The following is another example of a vendor or consulting company trying to make sales by showcasing their ranking in a Gartner Magic Quadrant or Gartner MQ.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: You are reading one of the only independent sources on Gartner. If you look at the information software vendors or consulting firms provide about Gartner, it is exclusively about using Gartner to help them sell software or consulting services. None of these sources care that Gartner is a faux research entity that makes up its findings and has massive financial conflicts. The IT industry is generally petrified of Gartner and only publishes complementary information about them. The article below is very different.

First, it is published by a research entity, not an unreliable software vendor or consulting firm that has no idea what research is.
Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services as a vendor or consulting firm that shares their ranking in some Gartner report. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department.

However, Sam Graham then asked the following question.

Naturally, this is not what the people at PwC who posted this on LinkedIn wanted. Instead, they wanted passive acquiescence to Gartner’s authority, even though PwC knows fully that they paid for their placement in Gartner. Without payment, they would not have scored well or may not have been profiled in the specific Magic Quadrant (MQ). As we cover in the article A Machine Learning Study of Gartner’s Magic Quadrant Bias, Gartner will frequently drop vendors or rated entities from its Gartner MQ from year to year — as if the entities they are rating no longer exist.

As we cover in the article How Gartner Research to Compares Real Research Entities we researched Gartner and compared them to real research entities. This included academic institutions, the RAND Corporation, and Consumer Reports. We found that Gartner met virtually none of the standards for a research entity.

The Reality Around Gartner’s MQs

There is actually no validity to any of the Gartner MQs. In the article How to Best Understand Gartner’s Business Model, we cover that neither Gartner nor the entities rated disclose their payments to Gartner now the amounts. PwC and Oracle could apprise us of their yearly “donations” to Gartner, but they won’t and will change the subject when asked.

This lack of transparency would not only completely negate the entity’s research credibility, but it is the exact model of entities that fake research and that, like Gartner, have their output prominently showcased in the vendor or industry advertisements, as we cover in the article How Gartner is a Fake Research Entity Like JD Power and Associates.

Research Problems with the MQs

There are several insurmountable problems with Gartner’s method in taking Gartner’s MQs as anything more than vendor-sponsored promotional material. Let us itemize just a few of these issues.

Issue #1:

Gartner receives around 1/3 of its revenues from vendors. However, at the same time, Gartner is rating these vendors. This is a direct conflict of interest. And there is a possible way to debate this point. If academic research followed this funding model, academic research would cease to come to accurate conclusions.

When Brightwork has analyzed vendors, we never asked vendors for money. Second, while there is supporting logic to charge vendors to access results (as studies do have to be funded), there is no logic for charging vendors for entry into a study. All manner for problems emanates from this model.

How does Gartner address the fact that some vendors have far more financial resources than other vendors and therefore can afford to pay far more money to Gartner? What one would predict if Gartner were using “pay for play” for its MQs is that the larger vendors would be highly represented, and the smaller vendors would be less so or entirely absent. And this is exactly what we find.
Gartner refuses to include open source in its MQs. Does this have something to do with open source projects not being able to pay Gartner?
The fact that vendors disappear from MQs and new vendors indicates that Gartner uses inclusion and exclusion from the MQs to make sure vendors pay to maintain their exposure. Gartner drops vendors from MQs from one year to the next, so the threat is real.

Issue #2: Gartner Does Not Disclose Payments or Amounts from Vendors

When Gartner publishes an MQ, nowhere is it mentioned that vendors paid to be in the MQs. We know that Gartner is paid because vendors tell us and because it is stated in Gartner’s annual report. Therefore we know the total amount of money that Gartner receives on an annual basis. So along with Issue #1, Gartner both has a financial conflict and attempts to hide this financial conflict and pretends they are not paid. This is straight research corruption that would disqualify a publisher if research rules were applied.

Issue #3: Gartner Uses an Ombudsman to Cover Up a Corrupt Research Model

When challenges on their model, Gartner will point to their ombudsman. This ombudsman is supposedly what ensures that all of the vendors get a fair hearing for their concerns. However, an ombudsman has nothing to do with research and is pulled from the media sphere. And one cannot use a non-independent employee to cover for Gartner’s lack of following research rules. A massive conflict of interest, and then deceiving ones clients about this financial conflict, are not remedied by a person called an ombudsman, who, as with the rest of Gartner, also functions with not transparency, and no publication of their decisions or even what vendors brought what claims and how this ombudsman ruled.

Issue #4: Gartner’s Faux Research Titles

Gartner employs analysts who often come from the most prestigious business schools in the country. However, virtually none of Gartner’s analysts have any research experience before being employed by Gartner. Yet, like a sales organization or a think tank that is really a front for industry (for instance, like The Heritage Foundation), Gartner gives out very elevated titles like “Distinguished VP Analysts.” These titles go out under emails and are used to introduced Gartner analysts and to give the overall impression that these analysts are accomplished when they are not. They are primarily people with a business education that have a job at Gartner. Using elevated titles like this is a tactic used by entities that are not performing serious research. This same issue applies to several other IT analysts in the space.

For example, Info Tech Research, (a company we caught plagiarizing our website and had them take down articles), has all manner of exalted titles for their analysts, but when you check their background before working at Info Tech, they would often have worked in procurement or marketing. How can a person have a career in marketing, and have an exalted research title by the fact of changing employment to Info Tech Research?

PwC also creates faux research titles as we cover in the article Why PwC’s Research Fellows are Fake and Pretend to be Academic. PwC’s “research fellows” do not work for a research entity, and their fellows report up through PwC marketing. When your research fellow reports through marketing, your research fellow is a marketer who rigs information to meet the demands of marketing.

Issue #5: Lack of Software Domain Expertise

The constant feedback from vendors is that they are unimpressed by the knowledge level of Gartner analysts. Most will have never worked on a software project and don’t understand the underpinnings of software. For example, Donald Fienberg’s work on SAP HANA illustrated that Denis demonstrated a low level understanding of databases, as we cover in the article How Gartner Got HANA So Wrong. This low level of understanding did not stop Donald Fienberg from having the exalted title fo “Distinguished VP Analyst.”

When reviewing many magic quadrants, the domain expertise in the areas where I can validate the expertise level by Gartner analysts appears to be low. Gartner emphasizes selecting employees who can communicate with executives, but this is not necessarily the right combination of skills to look for to perform analysis.

These are just five of the problems with Gartner’s MQs. There are many more.

Vendors pay GartnerCorruption of More than the Gartner MQ

We have repeatedly caught Gartner being paid by vendors to distribute what amounted to press releases for the companies that paid them under the guise of a “report.” Examples we documented include How Gartner Distributes Press Releases On HANA. Here is IDC doing the same thing IDC Takes Money to Publish SAP Provided Sample on S/4HANA.

That is, the problem is not restricted to Gartner. All of the major IT analysts function the same way. It is just that Gartner is the most prominent and most financially successful of them.

IT analysts do not do research as such — rather, they pretend to research and create marketing collateral for vendors and consulting firms.

Go to top

Conclusion

We know this because not only have we extensively analyzed Gartner, but we have had vendors reach out to us to create fake research for them.

They used different terminology such as “getting the word out,” etc.. But it was an offer of money for fake research. Furthermore, when we have been asked about our research, the vendor will make a statement like the following:

“We need to get some research that will show ABC, so that we can beat vendor XYZ.”

Therefore, the view that vendors have of research is that it proves exactly what the vendor needs it to. That is the conclusion is put before the research.

Vendors frequently seek to coopt or corrupt Brightwork Research & Analysis. Most vendors that reach out want free coverage or want to pay for flowery coverage. They will often attempt to leverage the reputation we have developed to provide accurate information to promote us to provide false information.

Gartner will provide exactly what vendors and consulting firms want — at a price. Vendors and consulting companies then prop up Gartner as a reliable source for research. It is a corrupt positive feedback loop.

Do Gartner’s MQs Have Any Legitimacy?