How do Complex Forecasting Methods Stack Up?
Executive Summary
- History shows some interesting insights as to complex methods.
- There are frequent misunderstandings regarding forecastability, which leads to the misuse of naive methods.
Introduction to Complex Forecasting Methods
Selecting the right forecasting methods is a critical component of forecasting. You will learn why sophisticated forecasting methods tend to be overestimated and how to choose the right level of complexity.
See our references for this article and related articles at this link.
Misunderstanding Forecastability
Individuals and companies and have a great tendency to overestimate forecastability. For challenging to forecast products, there is an interpretation that they can be overcome with more complicated methods. In finance very, complex models are trusted, which are often not much better than a guess. For instance, difficult to forecast products have weak demand patterns, and because of this, any statistical method employed has difficulty in predicting the future.
Underuse of Naive Methods
Naive forecasting methods, which should be used for many products, end up not being used because testing is often not performed, which compares the naive forecast to the more sophisticated method.
How the Incentives Work to Promote More Complex Forecasting Methods
Software vendors know that companies are looking for this magic bullet, and this is why a large amount of marketing effort goes into, including more advanced methods in the application. Similarly, academics have incentives to publish on sophisticated techniques because more complex methods are more publication-worthy and considered more prestigious than less complex methods.
As an example, it would be tough to get a paper published regarding the benefits of using a simple moving average because academic journals require that researchers submit “original work.”
The Effect of Computerization
There is little doubt; forecasting can be logically broken into pre-computerization and post-computerization. Automation was introduced at an elite level, so in the 1950s, researchers users computers to calculate mathematical methods applied to supply chain problems. Gradually computers enabled everything from warehouse workers to supply chain planners to do their job more efficiently.
George Plossl on Historical Inventory Planning
George Plossl had the following to say about historical methods for inventory planning:
“Planning was manual, slow and crude. Large clerical groups calculated gross planning requirements for major components of their products and time-phased (albeit very roughly) these and their procurement. Revising such plans was even more tedious and was rarely done. The capability of massive data storage and manipulation required for sound inventory management planning simply did not exist at that time. Because of this constraint, stock replenishment (OP/EOQ) methods predominated prior to the 1970s. Inventory control was attempted utilizing paper records and electoromachanical desk calculators to apply essentially simple mathematical formulas for order-quantity and safety stock calculations.”
Computers and the Methods Applied
As computerized forecasting methods gave way to more advanced statistical approaches that were encapsulated in software, there became a bias within software vendors to overestimate and over-sell the ability of their algorithms to forecast difficult to forecast products. For whatever reason, rather than introducing complex algorithms that were viewed as magic bullets by executives making the purchasing decisions, but often with little real-world experience performing forecasting or working as demand planners themselves.
How Was This Done?
This was often done while shorting the effort placed into developing either better demand planning interfaces or better data layers. This turned out to be a poor allocation of development resources. This is because of more advanced statistical models most often failed to outperform more straightforward methods. The infatuation with increasingly esoteric forecasting algorithms prevented executives from understanding that there was an upward boundary regarding how much a forecast for a product with little discernible pattern to its demand history could be improved. As pointed out by Michael Gilliland, a very reliable source of forecasting:
“Forecast accuracy is determined more by the forecastability of the demand than by the sophistication of a statistical technique. Statistical techniques work well as long as the historical patterns are properly captured in the forecasting model, there is not a lot of randomness in the patterns and these patterns continue to be the same in the future. Unfortunately, these conditions don’t always apply.”
Michael Gilliland describes the efficiency of statistical forecasting:
“A major benefit of the statistical approach is that it can be a very efficient way to generate forecasts. Once the system is implemented and the models are built, it can run on autopilot with little or not management intervention—which is about as efficient as an approach can get. In addition to being efficient, the statistical approach can also be effective in situations when demand follows consistent and detectable patterns, and continues to behave obediently into the future.”
Using complex methods as being usually more beneficial has been extended into machine learning and AI, where again, it is assumed more sophisticated methods are more effective. This leads to the topic of forecasting myths.
Significant Myths of Demand Planning
In comparing consensus-based demand planning to statistical demand planning within a historical context, I came across several myths that are believed by many but hamper effective decision-making on supply chain forecasting. I have listed them below:
- Only using statistical demand planning software would significantly increase forecast accuracy.
- More sophisticated statistical demand planning methods would improve the forecast over more straightforward methods.
- Consensus forecasting is where the majority of the opportunities lie to improve the forecast (versus statistical methods).
- Sales forecasting improves forecast accuracy because sales are “closest to the customer,” aka the more people that are involved in the forecasting process, the better the forecast will be.
The reason for the first and second myths was first given life was solely based on the incentives of software companies and consultants to sell.
Who Promoted These Myths?
These actors were able to convince the industry to invest in complicated software by the promise of better forecasts, but without ever actually proving that these more sophisticated solutions would improve forecasts. The third myth is related to the first two myths and can be seen as primarily a reaction to it.
Generally, it would be tough to find a consulting company that would provide the real story on these myths. Consulting companies have become so software-centric that they usually are partners with a software vendor, and repeat what the software vendor says. And most consulting companies are aligned with the largest software vendors, which generally have acquired their applications or built their applications without having many people that understand forecasting. Furthermore, marketing rules supreme at these vendors, so the people most experienced in forecasting don’t set the messaging.
Why These Myths Exist
These are myths because there was never any academic or other evidence to prove these ideas, yet they each at a different time gained widespread acceptance. One myth can lead to another myth, the essential element being that by believing something false, to begin with, it makes it easier to draw flawed conclusions a second time around.
To see the full screen just select the lower right-hand corner and expand. Trust us, expanding makes the experience a whole lot more fun
Conclusion
Multiple forecasting methods are often assumed to be more accurate than less complex simply because of the “wow” factor. Furthermore, more complex methods are less well understood by a larger percentage of the buyers of forecasting systems, and directors of forecasting, giving them what is often an unearned prestige.
A Better Approach
A primary reason that companies continue to be entranced by complex forecasting methods is that they do not repeatedly test how different forecasts compare against one another. And this is partially related to the problem of measuring comparative forecast accuracy.
Observing ineffective forecast error measurements at so many companies, we developed the Brightwork Explorer to, in part, have a purpose-built application that can measure any forecast. The application has a very straightforward file format where your company’s data can be entered, and the forecast error calculation is exceptionally straightforward. Any forecast can be measured against the baseline statistical forecast — and then the product location combinations can be sorted to show which product locations lost or gain forecast accuracy from other forecasts.
This is the fastest and most accurate way of measuring multiple forecasts that we have seen.