How Gartner Research Compares to Real Research Entities

Executive Summary

• Gartner positions itself as a research entity. However, compared with real research entities, Gartner performs poorly.
• We compared Gartner against academic research, RAND, and Consumer Reports to see how well they performed. Most people who read Gartner’s output will be surprised by how they scored.

Introduction (Skip if You Watched the Video)

Gartner is frequently referred to as a company that does research and is considered as such by those who purchase research and advisory services from them. However, aside from domain expertise, the following long-established rules make an entity a research entity. These rules are very well understood among those who perform research. For example, at Brightwork Research, we know and follow them. What often goes without mention is whether Gartner follows these research rules. You will learn how Gartner functions and how research, rather than commercial considerations, plays a role in how Gartner manages its business.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: You are reading one of the only independent sources on Gartner. If you look at the information software vendors or consulting firms provide about Gartner, it is exclusively about using Gartner to help them sell software or consulting services. None of these sources care that Gartner is a faux research entity that makes up its findings and has massive financial conflicts. The IT industry is generally petrified of Gartner and only publishes complementary information about them. The article below is very different.

• First, it is published by a research entity, not an unreliable software vendor or consulting firm that has no idea what research is. 
• Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services as a vendor or consulting firm that shares their ranking in some Gartner report. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. 

How Gartner Research Compares to Consumer Reports

I decided to evaluate Gartner’s research output by comparing Gartner’s rules and business practices to respected research rules. A comparison of Gartner’s research approach to that used by other analyst firms may seem like the long way around to understanding Gartner’s research output. Still, this chapter will significantly improve one’s understanding of Gartner’s research. While I have heard Gartner’s various criticisms, I have never seen Gartner compare against respected research entities on essential criteria. That is what is accomplished in this chapter.

One of the comparisons made in this chapter is between Gartner and the best-known rating company in the world: Consumer Reports. The second comparison is with one of the most respected think tanks globally—the RAND Corporation—which has established a reputation for impartiality and sometimes groundbreaking research for six decades. The third comparison is between Gartner and the academic research system, not with a single entity. Most US, European, and Australian/New Zealand research institutions subscribe. This chapter is essential as it sets the stage for later chapters. My research for this book highlighted that far too frequently, Gartner is discussed in isolation without the broader context of generally accepted research practice.

Gartner Versus Consumer Reports on Funding from Rated Entities

Consumer Reports is the US’s most trusted consumer rating and product-testing agency. Consumer Reports has been continually published since 1936. Consumer Reports has a very structured and well-designed set of rules that strictly limits the influence of those whose products and services are rated by them. Consumer Reports takes no advertising or other payments from those it reviews; instead, it is wholly supported by its subscribers and is the largest subscription-supported website in the US.[1] Compare this to Gartner, which takes money from those companies that it rates and actively solicits more business from current vendors who are customers. It solicits vendors who are not customers with what is generally known as aggressive sales tactics, which stop just short of promising better results in their ratings.

Consumer Reports takes no advertising or other monies from those they rate, which is critical. Their policy differs from that of JD Power and Associates, another well-known rating company. JD Power and Associates not only charge the vendors they rate but also charge vendors to advertise the JD Power and Associates award that they “won.” They rate the winners in a category but do not provide a complete list of the contestants, which is a nod to the vendors who did not perform well. Among a host of others, these factors are why JD Power and Associates is not seen as a severe research entity, and their principal usage is not by consumers but instead, as something that advertisers use to create the illusion of a good rating with consumers.

Gartner Versus Consumer Reports on Vendor Use of Ratings

Consumer Reports also has a non-commercial use policy, which means that companies rated well by Consumer Reports may not use the Consumer Reports rating in advertising or even on their website. Generally speaking, this is the opposite of IT analyst firms and Gartner. Gartner has some restrictions on how the ratings can be used, but the limitations are quite liberal. The last reference to the rules regarding how Gartner’s Ratings can be used can be found on the Office of the Ombudsman website.

Software vendors frequently publish their Gartner ratings (if they are good) on their websites, “tweet” them, and put them in press releases. So, software vendors advertise their Magic Quadrant rating quite aggressively if they have done well.

The Problem with Allowing Vendors to Use the Ratings in Advertisements (And Charging Them)

At first, it may not seem obvious why it should matter if those rated use the rating in their advertising. The reason relates to the perception of endorsement made by including a rating in such an advertisement and the strong tendency a vendor has to spin the rating to their ultimate advantage. Essentially, once vendors begin using rating results in advertising, Consumer Reports can no longer control the presentation of its results.  Consumer Reports explains their policy in the following way:

“At Consumer Reports, we believe that objective, impartial testing, reviews, and ratings are critically important for consumers. That is why we have a strict “No Commercial Use Policy” preventing the use of our name and information for any promotional or advertising purposes. The policy helps ensure we avoid even the appearance of endorsing a particular product or service for financial gain. The policy also guarantees that consumers have access to the full context of our information and are not hearing about our findings through the language of salesmanship.”

Gartner Versus Consumer Reports on Controlling for Sample Bias

Consumer Reports buys all of the products that they rate. This makes them confident that the products they rate are the same products consumers would receive. If Consumer Reports did not do this and instead accepted free samples, these samples would deviate from what was available to consumers. The manufacturers would ensure that only the best were sent to be rated. Not only do IT analyst firms not buy the software they rate, but they also don’t test the software they rate.

Ratings come from discussions with the software vendor, reviewing demos and questionnaires filled out by buyers, evaluating the literature produced by the vendor, and talking to companies that have implemented the software.

Gartner’s Lack of Testing or Even “Touching” of the Products They Rate

Therefore, Gartner differs significantly from Consumer Reports because they do not test the recommended products. When Gartner does see the product, it is a demo presented by the vendor in an artificial environment. Gartner uses a script during these analyst briefings, meaning that the vendor must show what Gartner has on its script and may not deviate from it. This is done to provide a consistent rating methodology to each vendor. Still, it has a disadvantage because it prevents the presentation of what could be exciting and helpful functionality. Several individuals who have participated in these analyst briefings have informed me through interviews that the demos are, in fact, lighter versions of customer demos, which are light compared to a demo that would be presented to a software-oriented person such as myself.

Gartner’s Statements on Improving Ratings Through Purchasing Advisory Services from Gartner

[1] Gartner’s sales group implies to vendors that they will improve their rating if they purchase consulting services from Gartner; this fact has been independently verified in multiple interviews with people who work in senior positions outside of the marketing department at software vendors. My confidence in this statement is reinforced by the fact that these individuals quoted Gartner’s same phraseology and that none of these individuals knew one another.

The idea that one would pay money for anything to a standard research entity and the potential for the rankings in the research to improve would eliminate any study from being taken seriously in the academic realm. Consumer Reports do not do it. The RAND Institute discloses its funding in each study that it performs, but it does not put itself in that type of conflicted position in the first place.

How Gartner Research Compares to The RAND Corporation

RAND is either the world’s pre-eminent think tank or one of the top think tanks, depending on who you ask. RAND was created over sixty years ago before think tanks had acquired their poor reputation. RAND is a “real” think tank, following—and often exceeding—academic research standards. Although they are highly productive, they employ only roughly 1,700 people. Upon occasion, I have read RAND’s research and was most impressed with their work quality. They were a significant innovator in software. I have written a book (Inventory Optimization and Multi-Echelon Planning Software), and their research was featured prominently in that book.

The RAND Corporation began its life primarily researching the Pentagon. Some of their best-known research has been in wargaming and strategies ranging from Vietnam (and from where the infamous Pentagon Papers were leaked) to the US anti-Soviet nuclear missile strategy. However, in the past few decades, they have diversified into a much broader research entity that researches everything from energy and the environment to health care. RAND also publishes an explanation of the standards they follow. I did not use all of the criteria published in this document because some do not relate to Gartner; RAND serves a public service function, whereas Gartner does not. However, several of RAND’s standards were universally applicable, and they are discussed and compared to Gartner’s standards in the following paragraphs.

Gartner Versus RAND on Referencing Past Work

RAND communicates very clearly how its research is performed.

“Although internal discussions about research quality have always been an integral part of RAND culture, more than a decade ago, we decided to codify in writing the quality standards for all RAND research. We intend the written standards to serve both as a guide for those who conduct, manage, support, and evaluate the research activities at RAND and also as the set of principles by which our research units and programs shape their individual quality assurance processes.”

RAND’s Standards for High-Quality Research and Analysis publication states how its research references other work.

“A high-quality study cannot be done in intellectual isolation: It necessarily builds on and contributes to a body of research and analysis. The relationships between a given study and its predecessors should be rich and explicit. The study team’s understanding of past research should be evident in many aspects of its work, from the way in which the problem is formulated and approached to the discussion of the findings and their implications. The team should take particular care to explain the ways in which its study agrees, disagrees, or otherwise differs importantly from previous studies. Failure to demonstrate an understanding of previous research lowers the perceived quality of a study, despite any other good characteristics it may possess.”

Gartner’s research is very much encapsulated and is much like Consumer Reports. A Gartner analyst might say that they don’t work in isolation; they talk to vendors and client executives daily. However, that is not the isolation to which this statement refers. This statement refers to other research unrelated to whether one accesses data sources. I write this as explicitly as I can, knowing full well that a Gartner analyst does comment on this criticism in some public forum. They will also mention that they are not isolated; therefore, I am mistaken and don’t understand how Gartner operates. However, I can only declare this several times and let the chips fall where they may.

Gartner Versus RAND on Transparency of Research Data

RAND’s position on the transparency of research data is taken from the same RAND publication as the previous quotation.

“Data and other information are key inputs to research and analysis. Data-generation methods and database fields should be clearly specified, and the data should be properly screened and manipulated. The research team should indicate limitations in the quality of available data. In addition, information presented as factual should be correct and verifiable.”

Gartner’s data is oftentimes not verifiable for the main reason that frequently, the data is not published. However, to meet RAND’s standards, the data from research must always be published.

Gartner Versus RAND on the Use of Jargon

It was interesting to see RAND take this position on the use of jargon:

“Necessary technical terms should be defined and explained.”

Gartner tends to use jargon in its writing. I recall one report that stated that a vendor’s application performed “stochastic optimization.” I was curious about how many people who read the report knew what the term “stochastic” meant! Gartner assumes that the readers know all the terms they use, but I do not recall seeing definitions at the end of their research. Although terms can be easily searched on the Internet today, it is probably less of an issue than it was in the past.

Gartner Versus RAND on the Use of Graphical Elements

RAND has this to say on the topic of graphical elements:

“To help explain complex and novel ideas, the documentation should augment textual exposition with graphical or pictorial elements.”

Gartner Versus the Academy on Peer Review

Gartner’s research is, to a degree, peer-reviewed before publication, but not with peers outside of Gartner. In academics, research is always peer-reviewed outside of the institution. According to Wikipedia: “Publications that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals.”

A lack of peer review can come across as a researcher having a particular bias or not being confident that the findings could survive a peer review. One of the best examples of the mistakes that can occur when research is not peer-reviewed is the example of cold fusion. In 1989, Stanley Pons and Martin Fleischmann, along with the University of Utah, submitted a research paper to the journals Nature and The TheJournal of Electroanalytical Chemistry —at the same time that they released a press release saying that they had discovered a way to create a nuclear reaction at room temperature. Pons and Fleishmann predicted to members of the media that:

“..cold fusion would solve environmental problems, and would provide a limitless inexhaustible source of clean energy, using only seawater as fuel. They said the results had been confirmed dozens of times and they had no doubts about them.”

The world was on its way to clean and limitless energy. There was only one problem: the results recorded by Pons and Fleischmann could not be replicated by other research teams that followed their methodology. The results, which led to the cold fusion rush, were eventually chalked up to poor research controls and concluded that Pons and Fleischmann had mismeasured their study. This research, which had not been peer-reviewed, led to several negative consequences, including millions of dollars spent in the subsequent decade—primarily by private industry—to replicate the faulty results. However, if Pons and Fleischmann and the University of Utah had followed the standard academic protocol of waiting to announce their results until after their experiment had been peer-reviewed and attempts had been made to replicate the results, the term “cold fusion” would not be in the popular consciousness. This is because there would have been nothing to report.

The Issue Gartner Would Face if Its Research Peer-Reviewed

Peer review works in the academic system because these institutions are part of a large ecology of universities that have worked this way for quite some time, and it is the accepted practice. It would be challenging for Gartner to have its research peer-reviewed. Who would review it: Forrester? These entities are all for-profit companies that do not share information with other companies— a significant limitation with research conducted by private entities. The research of Consumer Reports is not peer-reviewed; however, the transparency in its reports makes it auditable. Brightwork Research & Analysis is also not peer-reviewed. First, we provide much of our research for free, so we don’t have the resources to engage in a peer review process — and secondly, the IT area is very corrupt — and there are no peers that lack the financial bias that could engage in peer review.

I cover a very important foundational question regarding peer review in the article, The Problems With Peer Review Research.

Gartner Versus the Academy on Publishing the Methodology

The methodology used for research is such an essential part of the disclosure of scientific papers that it is part of the IMRAD acronym, which is commonly discussed in academics. This stands for:

1. Introduction
2. Methods
3. Results
4. (and)
5. Discussion

This is the standard order in which academic papers are presented.

Does Gartner Publish Its Methodology?

Several headings on Gartner’s website use the term “methodology.”

Some can be found below:

Supply Chain 25 Methodology

Gartner Methodology

These pages provide information about the methodology, as well as the research. Generally, at least to me, it is clear what each of Gartner’s research reports is measuring. However, as a person who has read a lot of research, I would classify Gartner’s disclosure on methodology as very light. Essentially, Gartner is misleading many of its subscribers (who do not have a research background), believing that this is a reasonable disclosure level.

For instance, let’s look at the methodology for the Magic Quadrant. The criteria used and which make up each axis are declared; however, are they all given equal weight, or are some weighted more than others? Gartner considers that disclosure of its methodology would not be considered a full disclosure by any of the research entities I have discussed up to this point. Gartner’s results, as published to subscribers, would never be published in any journal. Not all IT analysts follow this concealed approach. Forrester is one. Donald Ham explains how Forrester’s Wave (a competitor to Gartner’s Magic Quadrant) discloses much more to subscribers and allows subscribers to gain more value from the research.

“Forrester’s Wave product, their graphical vendor comparison tool, is similar in approach to the well-known Gartner Magic Quadrant.  However, the wave lets end users change relative weights of the comparison criteria by downloading and interacting with an Excel spreadsheet, resulting in a graphical display customized for the specific need­—a very cool enhancement.  Forrester is not as prolific with Wave reports as Gartner is with MQs, but where they exist for the products you’re investigating, points go to Forrester.”

Gartner’s Degree of Methodological Opacity Versus an IT Entity Like Forrester

The following table shows the criteria, the weights, and the data from Forrester’s Wave on BI Service Providers.

An excellent example of a far more complete disclosure of the methodology is available on the website for statistics for the Netherlands. I have included this link so readers can understand what I am describing.

ZL Tech’s CEO, Kon Leong, uses the following argument. The company referred to previously sued Gartner for damaging its business by unfairly ranking its products versus competitor’s products because of the company’s smaller size (its lawsuit was eventually rejected).

“The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores, so vendors know exactly where they are lacking and by how much and take corrective action.”

It is important to consider that Gartner’s target market is not researchers but decision-makers who are buyers, vendors, and investors. Many (but not all) of the people in these groups will be more interested in seeing the results. Many question how many people who read and rely on Gartner’s reports read the entire report, much less understand the research methodology. This gets back to the central theme throughout this book—that Gartner’s research is not particularly useful without the context provided by Gartner analysts. However, because the privately provided information is not part of the public record, it is also not auditable.

How the Methodology Is Interpreted

The methodology is one of the most overlooked areas of Gartner by those who read its research. The criteria are often not what one would expect to be used. If you speak with most people about Gartner’s methodology, they don’t know it. I can say this with confidence because when I tell interviewees what makes up the criteria for different analytical products, I frequently receive the response: “Is that what is counted by Gartner?”

Gartner has stated that readers are too quick to review a single graphic and not read the entire report. On the other hand, the way Gartner comments on its research is also responsible for providing the wrong impression regarding their methodologies, as I will demonstrate. I explain this in Chapter 5: “The Magic Quadrant.” However, for the sake of continuity, let me provide another example: Gartner’s “Supply Chain Top 25.”

The Gartner Supply Chain Top 25

Gartner publishes a “Supply Chain Top 25,” which most people would assume is the twenty-five companies with the highest-performing supply chains. This is the highest performance from the metrics commonly associated with supply chain management. These are forecast accuracy, inventory turnover, service level, and other similar measures). However, upon reading Gartner’s methodology for creating the Supply Chain Top 25, only one of these criteria is used. The criteria are instead:

1. Return on Assets (ROA) — Net income / total assets
2. Inventory Turns — Cost of goods sold / inventory.
3. Revenue Growth — Change in revenue from the prior year.
4. Gartner’s Internal Voting
5. Gartner’s Client’s Voting

Two criteria that determine the Supply Chain Top 25 have nothing to do with the supply chain but have to do with financial performance! However, when I read Gartner’s explanation of the research to media outlets, I found it did not match the methodology. The quotation below is an example of this.

“At the heart of the Supply Chain Top 25 is the notion of demand-driven leadership,” said Debra Hofman, managing vice president at Gartner. “We’ve been researching and writing about demand-driven practices since 2003, highlighting the journey companies are taking: from the old ‘push’ model of supply chain to one that integrates demand, supply and product into a value network that orchestrates a profitable response to ever-shifting changes in demand.”

Debra Hofman’s explanation is not the primary methodology of the study. Instead, the methodology focuses on the firm’s financial performance and people voting for who they thought should be on the list. As I’ve explained, the heart of the study is the methodology, not the notion of “demand-driven leadership.” The only control over whether a company showed “demand-driven leadership” is the one criterion Gartner voted on. The other four criteria could not have been related to this.

Furthermore, the definition of “demand-driven” is quite hazy. I work in the field and am unsure whether I have heard the term used before. Furthermore, the term “demand-driven” is not used once in the report. If it is such an essential component of the report, why is it not part of its methodology?

The Overall Scores

In the tables below, I have organized the scores of Gartner and each comparison entity. The tables have a few more criteria than those included in this chapter. These criteria were excluded from the text to keep the chapter manageable. The tables rank all entities against Gartner on every criterion:

Here you can see a description of how each entity performed in each criterion. Based upon this description, I have assigned a score, which is shown in the tables below: 

How the Scoring Works

1. The scoring methodology is that the higher the score, the better. The entity receives a ten if the issue does not apply to them. For instance, Consumer Reports does not receive any funding, so the issue of reporting industry funding does not even come up.
2. The zero and ten scores were simple; however, the scoring required some judgment or subjectivity when the criteria were not binary. Of the four entities, academic research was the most difficult to score because, while there are generally accepted research principles, there is considerable variability in their adherence. Biomedical research operates quite differently from other types of academic research and public versus private universities, among just a few of many variable factors.
3. Given RAND’s excellent reputation for impartiality and transparency, I was surprised that I could not find any record of financial contributions or a ranking of donors from large to small. In all the research entities I studied, it is difficult to determine precisely how much money each entity contributes. Although there is no question of which entity funds each of RAND’s research reports, and RAND lists its contributors (far more transparent than Gartner), this disclosure is still below what I would like to see. To see how transparent RAND is on this topic, I have included below an example of the statement incorporated on the front of each paper:

“This research is supported by the United States Air Force under Project RAND—Contract No. F44620-67-C-0045—monitored by the Directorate of Operational Requirements and Development Plans, Deputy Chief of Staff, Research and Development, Hq USAF. Views or conclusions contained in this study should not be interpreted as representing official opinion or policy of the United States Air Force.” – Descriptions of The Computer Program for Metric – A Multi-Echelon Technique for Recoverable Item Control

This clarifies who funded the study, even if we are not told how much funding was received.

Disclosure of Financial Contributions in the Academy

Academic research is far hazier in listing contributors, as the university contributors are never listed on the reports. Instead, the money tends to go into a big pot for each university department. But it would be difficult to doubt its corrupting influence. One of the more amusing contributions from the industry was the $100 million that Exxon Mobile paid Stanford to fund climate and energy research. One can imagine what type of “research” Stanford will be doing with this money and how many times this grant will be mentioned on Exxon Mobile marketing material.[1]

Because Consumer Reports is subscriber-supported, no one in the comparison group could match Consumer Reports in their lack of interest conflicts. I did not weigh the criteria as typically in a software selection matrix but considered each criterion equal in weight. The criteria should not be equally weighted, but I wanted to keep the table presentation simple.

Reasons for Gartner’s Low Score

The scores were quite interesting; as you can see from the tables on the previous page, Gartner scored relatively low compared to the other three research entities.

Gartner lost points for lack of transparency in several areas related to both data and conflicts of interest. I hoped the reader would have figured out that this is the type of report Gartner would produce. Now, Gartner analysts will not agree with the table and its findings. They may criticize individual criterion as not representative of good research practices (not a very good argument). They will likely declare the entire exercise meaningless because Gartner cannot be compared to the other research entities as they are unique. (Gartner would not score very well against other IT analyst firms. They would lose points against, say, Forrester because Forrester discloses their research data.). At least, these are the arguments that I would expect.

However, if we pause for a moment, it should be clear that what I have laid out in these tables has several advantages over the research presented by Gartner. Here is how my research differs from Gartner’s:

1. I explained the methodology for determining the description of each criterion per entity (this has been laid out in the comparison sections of this book).
2. The scoring methodology has been explained: it is based on the descriptions.
3. The complete research data set was shown, and anyone can see the scores for each criterion.
4. Although I have not yet made this declaration, it should be relatively apparent that I am not selling consulting services to Consumer Reports, RAND, or the general academic community. I have no conflict of interest that could pollute my results.

Therefore, while Gartner analysts may criticize the research at their leisure, the research adheres to a standard that no research produced by Gartner can claim to match.

[1] One of the things that made US universities the respected research institutions was that the public universities had a single contributor: the US government. The US government has a history of a long-term commitment to research without the necessity for a commercial outcome. Corporations cannot come close to matching this commitment. However, as the US university system is increasingly privately funded, conflicts of interest become more common. These quotations are from the Cal State University website:

“At a recent debate before Stanford’s Academic Senate, respected law professor Hank Greely argued that the tobacco industry ‘has perverted academic research for its own ends in ways that have had horrific consequences.’

‘It hurts me that my university gives them cover and sustenance,’ said Greely, a co-author of the resolution. ‘They are using us to whitewash themselves.’

For decades, university-based research funding was a strategic element of the tobacco industry’s effort to whitewash its tarnished public image.

Its disingenuous research was a focal point of a landmark trial last summer. On Aug. 17, 2006, Judge Gladys Kessler of the Federal District Court for the District of Columbia ruled that the tobacco industry had engaged in a 40-year conspiracy to defraud smokers about the health risks of tobacco. As evidence, she cited industry-sponsored work by UCLA epidemiologist James Enstrom, who challenged the view that second-hand smoke poses a serious health risk.”