How Much Can Your Forecasting MAPE Be Reduced?
Executive Summary
- Areas that roll into forecast accuracy should be separately measured.
- Forecast errors can be used to provide a direction for improvement.
Video Introduction: Forecast Improvement Estimation
Text Introduction (Skip if You Watched the Video)
There is a lot of poor quality advice on how to improve forecast accuracy. For example, software vendors tend to make the solution to improved forecast accuracy to be a new forecasting application. Those that want to sell AI/machine learning say the answer is more machine learning. The unfortunate fact is that what is “hot” in forecasting and what is pushed by vendors and consulting firms at a particular time often has more to do with a strategy to sell software and services rather than actually reducing forecast error. You will learn about some areas that are indisputably problems in forecasting departments, and that make sense to place effort to improve and that has a high probability of paying off.
Our References for This Article
If you want to see our references for this article and related Brightwork articles, see this link.
Relying on Forecast Error to Provide Direction on Improvement?
Merely relying on a forecast error is not a good indicator because, firstly, there are many questions regarding the error measurement. Secondly, what makes an acceptable, poor, or good forecast performance depends on as much.
Relying on a MAPE or other forecast error is not a good indicator. There are many questions regarding the error measurement. Secondly, what makes an acceptable, poor, or good forecast performance depends as much on the context (the product, type, industry, where the forecast is measured) as it does on the quoted error value.
Things to Consider
When considering the forecast performance or MAPE level of a company, what this comes down to is a series of factors that, when combined, result in the overall forecast performance. The industry or product type is not something that the company can control. However, many factors can be monitored, and if they can be controlled, then how well they are currently performing should be answered. A few of these are listed below:
- What is the performance of the forecasting system? (which is another combined question made up of the system’s capabilities, the match between the system and the company’s requirements, how the system is configured, how the system is integrated into the supply planning system or ERP system, etc..)
- What is the performance of the planners (that is how much, are the planners improving or degrading the forecast accuracy)
- What is the company doing to shape demand? (That is what incentives or disincentives are the company providing to customers which increases or decreases the forecastability of the company’s products)
Each of these areas needs to be analyzed and graded so that any business that wants to improve its forecast can understand where the most significant opportunities lie and prioritize where to apply resources for forecast improvement.
Segmenting the Factors that Contribute to Increasing Mean Absolute Percentage Error
The question to answer first is the forecasting system’s performance, out any manual intervention. If pure forecast system-generated output is not being measured against the adjusted forecast (after manual adjustment), then the company should begin doing this as soon as possible. Firstly, this allows the forecasting system output to be compared to another system. Secondly, computing the pure system-generated output to the final, or manually adjusted the forecast, can answer the second question, which is how much the planners are improving or degrading the forecast.
Conclusion
Companies have a constant desire to reduce MAPE, but in most cases, don’t take concrete steps to enable that improvement by first understanding where they stand in the different categories that contribute to the forecast performance of the system.
After using MAPE for several years, we finally moved away from it as a forecast error metric, as well as other standard error metrics.
Why Do the Standard Forecast Error Calculations Make Forecast Improvement So Complicated and Difficult?
It is important to understand forecasting error, but the problem is that the standard forecast error calculation methods do not provide this good understanding. In part, they don't let tell companies that forecast how to make improvements. If the standard forecast measurement calculations did, it would be far more straightforward and companies would have a far easier time performing forecast error measurement calculation.
What the Forecast Error Calculation and System Should Be Able to Do
One would be able to for example:
- Measure forecast error
- Compare forecast error (For all the forecasts at the company)
- To sort the product location combinations based on which product locations lost or gained forecast accuracy from other forecasts.
- To be able to measure any forecast against the baseline statistical forecast.
- To weigh the forecast error (so progress for the overall product database can be tracked)
Getting to a Better Forecast Error Measurement Capability
A primary reason these things can not be accomplished with the standard forecast error measurements is that they are unnecessarily complicated, and forecasting applications that companies buy are focused on generating forecasts, not on measuring forecast error outside of one product location combination at a time. After observing ineffective and non-comparative forecast error measurements at so many companies, we developed, in part, a purpose-built forecast error application called the Brightwork Explorer to meet these requirements.
Few companies will ever use our Brightwork Explorer or have us use it for them. However, the lessons from the approach followed in requirements development for forecast error measurement are important for anyone who wants to improve forecast accuracy.