How to Understand Selecting the Wrong Forecasting Software

Executive Summary

• Forecasting software is very frequently poorly selected.
• There are important implications for using incorrect fitting software to requirements.

Introduction

In speaking recently with a consensus sales forecasting software vendor, it was explained to me that in their space many of the companies they deal with often try to use utterly inappropriate forecasting solutions before they finally decide to turn to them. According to this vendor, the company in question first attempts to see if they can do the forecasting in their CRM system, then in their ERP system.

See our references for this article and related articles at this link.

Known to both me and this vendor, those systems are not suitable for the type of forecasting that the vendor I was discussing this with can very quickly.

The Broader Problem in Supply Chain Forecasting Software Selection

This discussion is reinforced by my personal consulting experience where I find, in most cases, companies using the wrong enterprise software to manage forecasting processes. One of the primary reasons for this is many decision-makers in this area are not familiar with the fact that the different types of forecasting all have specialized solutions now.

Secondly, trying to get a forecasting system that is designed to manage one process is very difficult to maintain an entirely different process. The various forecasting processes are represented by the diagram below.

The Biggest Mistake in Using Statistical Forecasting

By far, the most significant mistake companies seem to make is trying to use statistical forecasting packages to manage the other processes. Part of the reason that companies do this is because both the primary supply vendors (who tend only to offer statistical packages) and the major consulting companies (IBM, Accenture, Deloitte, etc..). Both tell companies that statistical forecasting software can be used to manage the non-statistical process forecasting. The reason the major vendors do it is evident that is what they have to sell.

The reason the major consulting companies do it is because they have built their consulting practices around the significant supply chain vendors, and their entire business model is based upon billing resources for major ERP vendors that can allow them to maximize their consulting dollars from the client. The concept of matching the best solution to clients needs is an entirely alien concept to them, and in 15 years of consulting, I have never seen a single major consulting company do it. In fact, due to their institutional structure, they can’t.

Forecasting Software Categories

The categories of forecasting software are as follows:

1. Spreadsheet Add-Ins (software in this category would be like CB Predictor)
2. Forecasting Modules of Statistical Programs (such as Minitab)
3. Neural Network Programs (such as NeuroShell Pred)
4. Dedicated Business Forecasting Programs (such as Time Series Expert)
5. Demand Planning Software

Unfortunately, they had to omit the final category and the category that would have been most interesting to me, because vendors would not submit their product for review.

Forecasting Principle Categories

The categories of forecasting principles are as follows:

1. Preparation of Data: This is how the data is manipulated before being imported into the application.
2. Method Selection
3. Method Implementation
4. Method Evaluation
5. Forecasting Accuracy Measurement
6. Assessment of Uncertainty: (Prediction Intervals)
7. Forecasting Presentation: (This was an interesting category they created, the authors describe this as helping forecasters in all phases of the presentation process, and having the ability to export data and forecast to end-users. They define critical areas of this category as forecast reports, user guides, and reference manuals)
8. Forecasting Across a Product Hierarchy: (this, of course, addresses top-down forecasting, but also reporting, etc.. I have found significant differences between the ability of forecasting products to manage hierarchies as I described in this article.)

 What They Found

They gave the lowest ratings to the spreadsheet add-ins. However, the other three categories (Forecasting modules of statistical programs, Neural network programs, and Dedicated business forecasting programs) scored higher with high and low vendors per area. Several of the top performers which I recognize were Auto box and Forecast Pro, as well as SAS/ETS, all three scorings at 70% or better.

Interesting Quotation

One of the most interesting quotes in the paper was about demand planning software vendors that denied their request for comparison and evaluation.

The paper has an interesting take on “why.”

Many companies are using forecasting modules that are part of larger demand planning supply chain management, or enterprise resource planning systems. These systems link forecasting engines with relational databases and with business application programs, and they are often sold with installation, training, and consulting services as complete forecasting solutions. The databases, business application functions, and individualized services can easily multiply costs over that of a forecasting engine by a factor of 10 or more.

Developers of these encompassing systems have sought to limit outside scrutiny of their software products, fearing that a negative evaluation or a comprehensive cost-benefit comparison might damage sales.

I think their observation is probably correct in some instances. However, demand planning software vendors probably don’t see sufficient benefit to doing this.

Also, many of the applications on the list are smaller and easier for a company to download, so would require less commitment on the part of the vendor. For instance, some of the software evaluated is Excel plug-ins.

One Caveat

The only unfortunate thing is that the research is at this point 12 years old.

Using Bad Forecasting Software

Most of the major vendors are promoting antiquated methods of data modelling, and this applies as much to forecasting as to business intelligence. These modelling approaches are costly to set up and to maintain. They don’t explain to their clients how it can replace the old and less efficient ways of data modelling, and they don’t care as it does not maximize consulting billing hours. This topic is involved but is covered in this article.

This is a significantly better way of performing forecasting. Yet not only do we have vendors recommending old methods based upon the legacy of the data designs of their applications that were developed some years ago — furthermore forecasting in applications like SAP DP which continue to perpetuate old methods but in very recently developed software. You can read about the details of the issues with SAP DP in this article.

Selling Bad Software

So many, unusually large vendors like SAP are using dated designs when it comes to forecasting and lacks sufficient numbers of people in their organizations that even understand forecasting.

This is why I spend time interacting with the consultants and developers in the best of breed vendors over the large vendors. Developing a forecasting system is more than merely taking a data modelling method and combining it with all the standard statistical methods.

The Problem of Leaving Out Important Requirements from the Selection Process

Requirements like forecast method and forecast error measurement tend to be quite emphasized, however many other elements are overlooked. They are, in fact, continually ignored because of the incomplete way in which the requirements percolate up to the decision-makers at multiple companies. Because of this, many vendors do not emphasize these requirements in their software development process.

One of these conditions is the historical adjustment. It is incredible how much custom development work must be performed and how poorly this development works in practice that is related merely to historical adjustment.

Consensus Forecasting vs Statistical Forecasting Methods

The problem goes beyond understanding how different forecasting systems are designed for specific forecasting processes. Currently, many companies are reading articles like the one I critique in this article, which describes a false dichotomy between consensus and statistical methods regarding what the company should focus on.

This article describes how Nike is moving towards consensus methods using a quote from AMR Research.

“There’s been a change in the technology for demand planning,” says AMR Research Vice President Bill Swanton, who declined to address the Nike case specifically. “In the late ’90s, companies said all we need is the data and we can plan everything perfectly. Today, companies are trying to do consensus planning rather than demand planning.” – CIO

The reason that the article in CIO is unhelpful is that Nike will always have a statistical forecasting process and a consensus forecasting process. Each of these processes requires software explicitly designed for the process in question.

Statements such as 85% or 60% of forecast improvement opportunities lie in (pick one) statistical, or consensus forecasting are not helpful. This is because as a rule of thumb to help direct effort because all of the forecasting processes must be addressed, and solutions exist to most efficiently address every type of forecasting process that the company faces.

Secondly, each solution can pay for itself in improved forecast error and lower supply chain costs if matched appropriately to the forecast process in question. Finally, to get a real understanding of specialized forecasting solutions, it is essential to move beyond merely categorizing forecasting software as either statistical or consensus.

Finding Better Alternatives

Many excellent alternatives are far better both at statistical forecasting than the major vendors, but also more appropriate for the non-statistical forecasting. Multiple solutions can be selected and blended as a solution at a far lower cost, and much higher business value than anything either SAP has.

Making the Case

In the previous post, I described the fact that most enterprise forecasting systems user interfaces are incredibly lacking. For how much companies spend on these types of applications, one would think that more inventiveness would go into the development of a powerful front-end for this class of software.

Areas of Differentiation in Demand Planning Applications

Except for advanced forecasting packages like SAS (which allow the user to customize the forecasting algorithms significantly), most vendors are offering mostly the same forecasting algorithms under the hood. (another unique offering is JDA DM with their Lewandowski Algorithm) The second area of differentiation, the backend data, is now commonly available in two different flavors, a relational star schema, or a multidimensional cube.

There are considerable differences here and how powerful and easy the back-end is to configure. This is a significant area the application which greatly determines is how robust and maintainable the application is. This is described in detail in this article. A related area of differentiation is how easy or difficult it is to get data into the application, which is covered in this article.  This leaves the user interface as a differentiator. However, the vendors seem to compete at all in the user interface hardly but are comfortable going to market with basically the same uninspired and confusing user interfaces that they have been producing for years. Two of the best interfaces are from ToolsGroup and Demand Works Smoothie. However, few software vendors come close to these two vendors, and most demand planning applications are saddled with weak user interfaces. This means that most companies that perform supply chain forecasting make their planners use lousy user interfaces. This, of course, dramatically reduces the ability to access the forecasting functionality within the application. The entire concept of interface usability is very rarely a focus on software selections.

Conclusion

Companies are wasting enormous amounts of money on forecasting solutions that have a poor implementation history. It is sporadic that companies select the right forecasting system for their needs and are too often limited to using just one forecasting application. They have blinders imposed on them by major vendors and the large consulting firms causing them to continually apply statistical forecasting solutions that require different designs, for which there are ample alternatives in the marketplace.