How to Understand Gartner's Mistake on Big Data Forecasting

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

This article by Gartner makes several fallacious statements about forecasting and Big Data.
We analyze the problems in Gartner’s understanding of forecasting.

*Note, the article mentioned here was a reprint of Gartner’s content by a company called BlueRidge. The article was republished, making it challenging to determine if the material was created by BlueRidge or by Gartner. It was the objective of BlueRidge to make it seem that its content was Gartners’. Therefore, the statements below are from BlueRidge, not from Gartner. I would suggest that Gartner not allows companies to “republish” their research as it confuses them as to what is Gartners’ and what is BlueRidge. It is also difficult to believe that Gartner was not at some point made aware of the additions by BlueRidge to Gartners’ content.

Introduction

As probably most of you have, I have been a follower of Gartner for some time. They are so influential and so often mentioned within companies that I wanted to peek behind the facade to see how they “make the sausage.” I researched how they develop their rankings and how they research the book Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors. This is the only book written by an outsider to Gartner. I did not have to worry about offending any contacts at Gartner, so I analyzed their methodology objectively, how their ombudsman works, and generally how they do business.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Go to top

The Gartner Story

Gartner is a fascinating story and a great marketing company to operate in the enterprise software space. I don’t know anyone in the space that rivals them in marketing prowess. Gartner, at a high level, works in the following way.

Gartner is, in its essence, an information broker. They get paid talking to software vendors (and collect information from them) and then turn around and get paid to speak to software buyers (and gather information from them). Brilliant!
In part, Gartner performs analysis and then sells software information to buyers and buyer information to software vendors.
Gartner is a money machine. However, they do not declare which and how much different software vendors pay them in breaking with fundamental research principles.

Gartner’s Statements on Forecasting

While I could go on about Gartner all day long, this article will focus on recent statements made by them in their most recent Magic Quadrant on Supply Chain Planning for 2016.

Gartner made predictions that I will address in this article that I am as close to positive as I can be are incorrect.

Customer Forecasting: The first prediction is that Big Data will switch forecasting from the product and more towards the client.
Causal Forecasting: The second prediction is that Big Data will lead to far more causal forecasting.

So let us begin with the first topic by reviewing the direct quotation.

“Nearly everyone talks about the item’s demand, and building a forecast based on what the item is doing. But what if you’ve been forecasting the wrong thing? The fact is, items don’t really “do” anything. They don’t sell themselves. They don’t make decisions. They don’t really even have a history. It’s the customer who has history, who buys, who makes decisions, who influences your stock levels. Any statistics you have on the item does not derive from the item, but from the customer. The customer’s behavior is what you should be tracking. But until the era of Big Data and sophisticated analytics, tracking and predicting behavior was not possible. Big Data and the ability to analyze customer transactions have revolutionized the understanding of customer demand, providing visibility and precision on a whole new level.”

The best way to describe this line of thinking is that it is a high concept, but the most accurate way to describe it is nonsense.

The Big Data Bubble

Big Data has become a sexy new concept, and now it is seen as the magic elixir for a wide variety of problems. During a pause in a presales presentation, all one has to do is utter something non-specific like “it’s Big Data,” or “it’s all about the Big Data,” and it is just so hard for the audience not merely to nod and approve. This marks the existence of a bubble.

Conferences are awash with energetic optimism regarding the potential benefits of Big Data. This has become a massive buzzword among CEOs, who may not even know what Big Data is. This allows a person to say things that don’t make any sense but seem visionary, one reason being there is so much other hype around Big Data that almost any comment can seem reasonable. I recently tried to push the bubble myself.

I stated to someone that “Big Data is going to revolutionize laundromats because it will bring deep analytics to who the customers are, and it will all be happening in real-time.” It seemed to work. They agreed or at least did not protest. You can try this experiment yourself.

Why Gartner is So Wrong on This Topic

Let’s get into the detail of why I believe Gartner’s statement regarding transitioning from forecasting products to gaining deeper insights into a customer is incorrect.

People who often like to talk about changing the forecast to forecast at a different attribute will often lose sight of supply chain forecasting’s fundamental requirement.
The company is stocking products, and it, therefore, must (at some point) generate a product forecast for inventory management to work correctly.

As with supply chain forecasting systems, supply planning systems produce a supply plan in the form of goods and quantities to be ordered. This means a product forecast at a location. The relevant factor is over replenishment lead time (let us hold off on forecasting lead time as it goes how the forecast accuracy is measured, the value to supply planning, and generally into another layer of complexity).

Big Data forecasting does nothing to change this foundational rule of supply chain planning.

The Complication of Customer Forecasts

Sales and Marketing perform forecasting by the customer. However, it does nothing for supply chain planning. No matter what hierarchy Sales and Marketing chooses to use to produce its forecast (by product group, by sales group, by region, etc…), the supply chain forecast must be of the product, and more specifically at the product location combination.

Eventually, the Sales/Marketing forecast must be disaggregated to a product location, (which I cover in quite a lot of detail in the book Sales and Statistical Forecasting Combined).

It should be acknowledged or at least understood that the primary reason that sales create forecasts by a customer is that they use their knowledge of customers to make manual adjustments to the forecast. Note that in very many situations is the customer being used to generate a better “automated” forecast.

The Normalcy of Using Customer Data

Customer data and its association with products have been used in forecasting applications for quite some time.

Any forecasting application, which can apply attributes or hierarchy, can use the customer as an attribute to perform a top-down forecast, which can then be analyzed and then used to drive the customer’s influence down to the lowest level of the forecast hierarchy.

The effect can be tested against other attributes (color of the product, product group, etc..), and then the impact of each attribute can be measured.

Big Data Forecasting’s Involvement? None of this has anything to do with Big Data forecasting. I do this work comparing customer attributes versus other attributes in a forecasting application without Big Data’s involvement (as explained in this article).
The Customer Ship to? Forecasting at the client is the general term. Still, in many cases, a company has multiple ship-to-locations for a single customer. Therefore forecasting at the customer ship-to-location is another option. (as explained in this article.) Sometimes people who have a poor grasp of supply chain forecasting will recommend switching the product to the customer or customer ship-to-location. The company then begins thinking regarding forecasts at the customer or customer-ship-to, thinking that the change is improving forecast accuracy. This does not work and requires lots of adjustment to get the forecast eventually back to a product location level, which is used by supply planning.

A Conspicuous Lack of Evidence for Gartner’s Claim for Big Data Forecasting

Gartner is making a very bold claim that Big Data will transform forecasting to be far more customer-based. Still, no references or other evidence are provided that Big Data is currently being used to improve customer forecasts.

So, I searched on a comprehensive academic paper database, and I found 28 total results from the search Big Data Customer Forecasting. However, upon review, most of these documents were not real hits on the topic but false positives. I have been researching issues for quite some time, and I can say it would be strange for there to be no or close to no articles on a topic in academics but for Gartner to be writing on the fact that this is either happening or very close to happening.

Therefore, in my view, Gartner’s prediction on customer forecasting with Big Data is off, but Gartner was not finished proposing how Big Data forecasting.

They next moved on to making another prediction regarding something called causal forecasting. This is very similar to the forecast we just covered in that it sounds very enticing, particularly to people who are less familiar with forecasting. Let’s take a look at that because it shows Gartner’s pattern referencing Big Data forecasting improvement.

Gartner and their Prediction on Causal Forecasting

In the most recent Magic Quadrant on Supply Chain Planning for 2016, Gartner made the following peculiar statement on the nature of the opportunity regarding Big Data forecasting:

“In particular, identifying influencers that impact customer behavior can move you closer to the holy grail of predicting the customer. Every single customer transaction is influenced by causal factors. The truth is that causal simply identifies what causes, or influences a shopper to buy something. When you know “why,” you have a much better chance at accurately forecasting “what” “when” and “how much.”

Gartner Versus the History of Causal Forecasting

This paragraph is problematic from some dimensions. The first of which being that predicting the customer is not the Holy Grail of forecasting. But to get to the meat of it, Gartner proposes that Big Data forecasting will allow forecasts to be created which are causal, which is somehow new to forecasting.

So what is causal forecasting, to begin with? Well, causal forecasting is where there is an attempt to identify an independent variable predicting the dependent variable.

Using Causal Forecasting

So if you run a lemonade stand and notice that sunny days outsell overcast days, you can use the weather forecast to plan when you will work and when you will take the day off.

First, causal forecasting is not at all new to supply chain forecasting. Quite the contrary, the ability to create causal models has existed for many years in many supply chain forecasting applications. There are a virtually unlimited number of academic papers written on the topic and the areas where causal forecasting is applied are much more significant than merely supply chain forecasting.

In fact, in the supply chain, forecasting causal models are very rarely used regardless of the availability of causal forecasting functionality. Gartner’s implied assumption that the limiting factor in using causal forecasting has been not having access to Big Data is not correct. Here is why:

Unrelated Limitations to Causal Forecasting: The definition of Big Data is where the data sets that are processed are so large or complex that traditional data processing techniques are inadequate. This means using applications like Hadoop, NoSQL, and so on to manage enormous amounts of often unstructured data. However, processing massive quantities of evidence using non-traditional processing techniques has never been the limiting factor in creating causal models.
Actual Limitations in Causal Forecasting: The limiting factors have ranged from not having access to causal factors (because the data is not maintained or is not of sufficient quality to use) to not having the time or expertise to build causal models.
When Causal Forecasting Tends to be Used: Causal models are often used where the number of forecasted items is small, and the financial benefit (or assumed financial interest) to forecasting is enormous. An excellent example of this is forecasting in the financial services industry, where investment banks have few forecasted products and big budgets. Big Data does nothing to address the limitations that have caused causal forecasting to be infrequently used in supply chain forecasting.

Gartner’s Influence and The Reaction to Their Observations

Gartner is hugely influential, so what they say matters. So it matters, but it is important not to confuse matters with being correct. Here Gartner provides verifiably incorrect information to its clients, which has a strong potential to lead their customers down pathways that are resource and time-wasting.

Many people will assume that because Gartner is saying it that it must be true.

Is Gartner Simply Pumping People up on The New Sexy (Big Data Forecasting)?

Gartner’s proposal is implausible to add to forecast accuracy, but is it only said to be topical, or do Gartner’s analysts believe what they are writing?

In this case, the unfortunate thing is that companies significantly underuse the functionality available to them in the forecasting applications that they have already purchased. They so often end up with inappropriate forecasting software because they don’t know how to perform software selection that I covered years ago in this article.

Distractions Galore (Big Data Forecasting Being One)

Companies already have significant issues in implementing and maintaining their current forecasting applications.
The investment made by businesses that improve prediction accuracy is low, particularly in proportion to what random forecast error costs businesses in operational inefficiencies, waste, etc..
Companies, in most cases, are not applying well-established and tested approaches for improving forecasting accuracy.
There are exceptions, but the knowledge of forecasting fundamentals in businesses is weak, making many of them particularly susceptible to the erroneous information of the typewritten, in this case by a credible source like Gartner.
Focusing on Big Data forecasting will almost certainly disperse the limited attention, funding, focus they do have, and away from actually improving their forecasting applications or choosing better ones.

Go to top

Conclusion

Following Gartner’s advice on investigating Big Data to improve forecasting accuracy moves companies away from focusing on the real solutions available to them and is a distraction that has an amazingly low probability of improving forecast accuracy.

The Impact of Listening to Gartner on These Topics

Those in the position to heed Gartner’s advice on significant data forecasting should be wary.

Gartner’s information in these two quotations is incorrect, and Gartner may or not believe it themselves (I will cover why in a future article). But it is clear Gartner is making up quite a bit about big data forecasting.

Gartner’s proposal regarding forecasting at the customer will result from not only not improving the forecast; it will quite to the contrary, reduce its accuracy.