Forecast Error Myth #1: One Can Rely on The Forecast Error Measurement in Forecasting Applications

Executive Summary

• Companies tend to think that forecasting applications manage their forecast error measurement.
• We cover the issues with relying on forecasting applications for forecast error.

Video Introduction on Forecast Error Myth #1: One Can Rely on The Forecast Error Measurement in Forecasting Applications

Text Introduction (Skip if You Watched the Video)

Companies that purchase forecasting applications prefer to think and believe by software vendors and consulting firms, that all forecast error measurement is taken care of in their applications. Nearly all companies face that own forecasting applications continue to have forecast error measurement problems after years of using these applications. This is because most forecasting applications are only designed to provide a forecast error calculation at the product location combination. You will learn the reality of what forecast error measurement is provided in most forecasting applications.

Our References for This Article

If you want to see our references for this article and related Brightwork articles, see this link.

Myth #1: The Myth of Effective Forecast Error Measurement in Forecasting Applications

This is one of the major myths we have identified about forecast error measurement in the forecasting area. It is largely unquestioned, even though it can be easily disproven.

The myth is so durable because of a misunderstanding of what forecasting systems do and how they are designed.

For All of the Myths, See the Following Table

The Design of Forecasting Applications

Forecasting applications are designed primarily to do the following:

1. To produce forecasts
2. To match different forecast techniques to the demand pattern (normally called best fit)
3. To allow users to make manual updates overrides.
4. To allow the forecast system to hold multiple rows, representing adjustments and forecasts, all of which can influence the final forecast and provide visibility and traceability.

What Forecasting Applications Normally Provide for Error Measurement

Forecasting applications will nearly always provide several different forecasting calculation measurements, but all at the product location combination. The calculation may provide a value when a group of products or locations is selected, but the error has no meaning, as it requires some weighing to be performed. An additional issue is that while forecasting applications use the major forecasting error measurement formulas, there is an implied assumption that each of the error measurement formulas effectively helps forecast improvement.

Reporting Out Forecast Error from the Demand Planning Department

The forecast error measurement in forecasting applications is primarily for the user or planner.

Companies will then create a custom report that pushes the forecast error, or planners will export the error to Excel, perform the calculation in a spreadsheet, and provide it to their Director of Forecasting/Demand Planning, who then provides it to the broader company.

Conclusion

Forecast error calculation is misunderstood comprehensively within forecasting applications, even though those outside of the applications’ users never see this calculation.

A second process delivers the forecast error that is the forecast error that is actually tracked to the broader groups of stakeholders.

Why Do Companies Have So Many Problems With Forecast Error?

While there are several reasons that we have documented, our view is that this myth that the forecasting system produces a usable forecast error is one reason for this disconnect.

Our further view is that forecast error calculation in companies by in large does not help companies drive to improved forecast accuracy. We have demonstrated this to many companies by asking those that receive the forecast error report what they think it means and what they do with it. We found the forecast error report invariably is a measurement that desired to be seen, but which is so misunderstood and so misrepresented in its calculation versus what the multiple recipients think it is, as to be useless.

What the Forecast Error Report Does Not Do

1. The forecast error does not show the recipients what different forecast models’ applications can improve product location combinations.
2. It does not tell the recipients which forecast inputs are valuable and which reduces forecast accuracy. Furthermore, it promotes a dysfunctional approach to forecast management.

Why Do the Standard Forecast Error Calculations Make Forecast Improvement So Complicated and Difficult?

It is important to understand forecasting error, but the problem is that the standard forecast error calculation methods do not provide this good understanding. In part, they don't let tell companies that forecast how to make improvements. If the standard forecast measurement calculations did, it would be far more straightforward and companies would have a far easier time performing forecast error measurement calculation.

What the Forecast Error Calculation and System Should Be Able to Do

One would be able to for example:

1. Measure forecast error
2. Compare forecast error (For all the forecasts at the company)
3. To sort the product location combinations based on which product locations lost or gained forecast accuracy from other forecasts. 
4. To be able to measure any forecast against the baseline statistical forecast.
5. To weigh the forecast error (so progress for the overall product database can be tracked)

Getting to a Better Forecast Error Measurement Capability

A primary reason these things can not be accomplished with the standard forecast error measurements is that they are unnecessarily complicated, and forecasting applications that companies buy are focused on generating forecasts, not on measuring forecast error outside of one product location combination at a time. After observing ineffective and non-comparative forecast error measurements at so many companies, we developed, in part, a purpose-built forecast error application called the Brightwork Explorer to meet these requirements.

Few companies will ever use our Brightwork Explorer or have us use it for them. However, the lessons from the approach followed in requirements development for forecast error measurement are important for anyone who wants to improve forecast accuracy.