How to Best Understand the Naive Forecast
Executive Summary
- The naive forecast creates a baseline forecast that can allow a forecast value-add for more advanced methods.
- Why the naive forecast so underused.
Introduction
The naive forecast creates a baseline forecast that can allow a forecast value-add for more advanced methods. Why the naive forecast is so underused will be discussed as well. According to Steve Morelich, the naive forecast (the same as last period) beats more complex forecasts in up to 50% of the product database. That is a fantastic conclusion, and this is primarily unknown by forecast practitioners and usually is not discussed very much by software vendors or consulting firms. A common reason is that software vendors and consulting firms want to continue to present the fiction that their software and approaches contribute significant value over the naive forecast. You will learn why testing your forecast against the naive forecast is an essential first step in evaluating forecasting effectiveness.
Our References for This Article
To see our references for this article and related Brightwork articles, visit this link.
This fact is largely unknown by forecast practitioners and is usually not discussed very much by software vendors or consulting firms. A common reason is that software vendors and consulting firms want to continue to present the fiction that their software and approaches contribute significant value over the naive forecast.
The naive forecast is the starting point for statistical forecast improvement. However, improving the accuracy by overusing the naive forecast is tricky for many product location combinations.
Answering the Basic Questions for the Naive Forecast
The naive forecast is named as such as the forecast is unsophisticated.
- Although it is sometimes defined as merely the last period’s demand, a naive forecast is not a distinct demand planning model.
- A naive forecast can be several models, all of which are simple in their design.
- A naive forecast can simply be the sales from the last period, a moving average, or, for seasonal items, what was sold in the previous year in the same period.
Why is The Term “Naive Forecast” Viewed as a Negative?
Some use the term “naive forecast” as a derogatory term. A naive forecast can be surprisingly effective.
Question
If a naive forecast is so unsophisticated, why would it be used?
Answer
The answer is that naive forecasts can often beat demand planning models which are much more sophisticated and higher cost to implement.
Most companies never create a naive forecast and do not perform the second step of comparing their current forecast against the naive. Consequently, these companies do not know the value add of the forecast they are creating.
The Argument for Creating a Naive Forecast
Michael Gilliland at SAS is a proponent of creating a naive forecast for every product and comparing it to the company’s forecast to estimate the forecast value added to the demand planning process. The idea is that the forecast accuracy improvement shows the benefit of using more complex methods and putting more “energy” into the forecast. However, it is essential to measure this because many forecast accuracy improvements that put more energy into the forecast don’t necessarily improve the forecast or are not worth the extra energy to create the forecast.
By testing the more energy-consuming forecast against the naive forecast- which, by definition, takes minimal effort- the “value add” of any forecast can be roughly determined.
The objective is to find the least energy-consuming forecasting approaches that provide the greatest possible return or the highest forecast value addition.
The Lack of Interest by IBM and Many Consulting Companies in Measuring Forecast Value Add
Many companies don’t want to know the value added to their current forecast over the naive. For example, IBM currently has sold many machine learning forecasting projects globally. IBM does not want its machine learning forecasts tested against a naive forecast because it would quickly call into question the value of the project that it sold. As we cover in the article How Many AI Projects Will Fail Due to a Lack of Data?, IBM measures its success not by the benefits of its machine learning project outcomes but by its ability to sell more machine learning projects.
Keep the Outcome a Secret
Many projects, including forecast improvement projects, are sold without considering the outcome. The intent is to make exaggerated claims and then stall the project’s measurement of outcomes to accumulate the maximum number of billing hours.
If your forecast improvement project is sold based on pretenses, you do not want the forecast value to be measured. For this reason, IBM postpones forecast error measurement on their projects as long as possible, preferring to talk about how significant the INPUTS are and the tremendous future potential for forecast improvement.
How Much Do Companies Normally Know About Their Statistical Forecast?
In a recent article, I described how companies don’t know how much they can improve their forecast accuracy. In that article, I covered the fact that businesses find themselves in this position because they do not distinguish in their forecast accuracy measurements between manual adjustments and the system-generated result.
There are other reasons for this problem as well.
Suppose we limit the discussion to just the statistical forecast. In that case, I have observed that companies also don’t know how much the forecasting models they are using improve or degrade the forecast over the simplest model they could use. If they don’t know this, they cannot know how much extra effort they should put into more complex forecasting methods.
What is a Naive Forecast?
This simple forecast model is called an original forecast. A naive forecast can be simply the sales from the last period, a moving average, or seasonal items sold last year in the same period.
Performing a naive forecast and comparing it to the accuracy of the currently generated forecasts is an excellent approach to understanding how much value is being added to the current forecasting process.
What Should One Expect from Testing Against the Naive Forecast?
The results of a simple forecast comparison are often surprising.
When the naive forecast is tested against experts in the financial industry, the naive forecast often wins. In the History of Forecasters’ research paper, the expert consensus opinion was compared against a naive forecast. Surprisingly, the very expensive Wall Street experts used in this comparison lost out to the naive forecast.
A quote from this research is listed below:
“Our analysis of semi-annual Treasury Bond yield forecasts as presented in the Wall Street Journal shows that the consensus forecast is poor. Over the past 43 forecast periods, the consensus estimate of the yield change has been in the wrong direction 65% of the time. A naive forecast of the current yield results in 11% reduction in the standard derivation of forecast error.”
It has been found that financial analysts cannot beat naive forecasts when taken as a group. This is one of the primary arguments used to develop index funds and why Vanguard became such a household name in investing and retirement savings.
The Political Dimension to Socializing Naive Forecast Accuracy Comparison Results
Michael Gilliland of SAS proposes that beating a naive forecast is more complicated than most people assume.
Also, be aware that naïve forecasts can be surprisingly difficult to beat. When you report your results, they may be rather embarrassing to those participants who are failing to add value. Therefore, present the results tactfully. Your objective is to improve the forecasting process – not to humiliate anyone. You may also want to present initial results privately, to avoid public embarrassment for the non-value adders.
As Michael observes, people don’t like admitting there is no value to their actions, so they reflexively push back on the idea that the naive forecast can work better than their models. If a person is paid to do something and does not add value, they will hide the outcome from any verifiable measurement. This happens all the time — and the audience is required to “take the proposer’s word for it” and that they have (INSERT # OF YEARS) doing something — therefore, the outcome must naturally be good.
When To Use the Naive Forecast?
If a forecast model cannot beat the naive forecast, or if it can, but the effort put into creating the naive forecast is not worth the improvement, then the naive forecast should be used.
Comparing a method against the naive forecast is how a forecasting method is determined if it adds value. For instance, many people are very optimistic about Crostons as a method for improving lumpy demand forecasting. However, some tests have shown that while Crostons is more accurate than more simple methods, mainly when including minimum lot sizes on the supply planning side, Croston’s may not add as much value as originally thought when looking at the forecast accuracy improvement.
How Long Should You Test the Naive Forecast Against the Current Live Forecast?
Michael Gilliland makes the good point that one should give it time when evaluating a forecasting step.
The FVA approach is intended to be objective and scientific, so you must be careful not to draw conclusions that are unwarranted by the data.
For example, measuring FVA over one week or one month does not provide enough data to draw any valid conclusions. Period to period, FVA will go up and down, and over short time frames FVA may be particularly high or low simply due to randomness.
When you express the results in a table, as we’ve shown up to this point, be sure to indicate the time frame reported, and make sure that time frame has been long enough to provide meaningful results.
However, while this is necessary for many improvements made to the forecast, within statistical forecasting, it is possible to turn the clock back and make a forecasting application think that it is six months before the current date.
Effectively Turning the Clock Back
When this is performed, a naive forecast can be compared against the company’s forecast versus the actuals generated to determine if the naive forecast performed better.
- However, the differentiation between system-generated and naive forecasts often cannot be accomplished.
- The manual changes are saved as the “final forecast.”
When this comparison is being made, unless the system-generated forecast is known, this test will not tell you if the naive forecast was inferior or superior to the system-generated forecast, and that is a problem.
However, Gilliland’s recommendation regarding being scientific about the measurement and being sure before switching from the current forecasting method to the naive is quite right.
System Implications of the Naive Forecast
To perform a naive forecast, the application must be able to create a forecast and store it in a location that does not interact with the final forecast. This naive forecast can be created offline in the production (in use/live) application without affecting the live forecast. For instance, in most forecast applications, any forecast can be generated and stored in a measure/row (which is not the final forecast).
Therefore, this forecast is kept separate from the forecast (called the final forecast), which is sent to the supply planning system.
Is Naive Forecasting An Underused Forecasting Approach?
Yes. Many companies resist using the naive forecast, even in areas where it would provide a higher forecast accuracy or a higher effort/energy-adjusted forecast accuracy.
- Somewhat surprisingly, naive forecasts are significantly underused. This is primarily due to political barriers.
- It is hard to sell the use of naive estimates as the general assumption is that more mixed methods can always beat simple forecasts. This extends across different forecasting areas, and in every field, naive methods tend to be rejected for more complex methods. This is described in the history of complicated versus simple methods.
Why the Naive Forecast Became Popular in Investing but Not Supply Chain Forecasting
As we discussed earlier, the naive method is used by many without their knowledge. The index mutual fund is a naive forecast strategy and is the opposite of active stock selection. This shows that much of what makes a method acceptable or popular comes from its naming.
However, the index fund became famous only because of an interest in measuring the outcomes. In many companies, there is less interest in measuring what works than in justifying the investments and the strategies that the senior management has already selected within these companies. It should also be noted that even in investing, forecasts that do not beat the naive are repeatedly referred to, widely disseminated, and respected, even though there is no evidence to beat a naive forecast.
Comparative Forecast Error Measurement
To test the value added of using a non-native forecast, one must be able to measure comparative forecast accuracy efficiently. As we cover in the article Forecast Error Myth #5: Non-Comparative Forecast Error Measurement is Helpful, there is a potent myth that one does not need to perform comparative forecast error. And related to this myth is a second myth that forecast error is effectively measured in forecasting applications, as we cover in the article Forecast Error Myth #4: Most Forecast Error Measurement Supports Identifying How to Improve Forecast Accuracy.
As companies tend to lack an automated way of performing comparative forecast error measurement, there is often little understanding of how much other forecast methods (or manual overrides, marketing adjustments, etc.) improve or degrade the forecast error.
Conclusion
- It is essential to “baseline” the forecast by performing a naive forecast for all products. This lets the company understand how much value is added to the current forecasting process.
- It also helps provide an impression of how difficult the products are to forecast. When a better forecasting method is applied, how much value is added to the forecast can be understood.
All of this requires the ability to measure comparative forecast accuracy efficiently.
Why Do the Standard Forecast Error Calculations Make Forecast Improvement So Complicated and Difficult?
It is important to understand forecasting error, but the problem is that the standard forecast error calculation methods do not provide this good understanding. In part, they don't let tell companies that forecast how to make improvements. If the standard forecast measurement calculations did, it would be far more straightforward and companies would have a far easier time performing forecast error measurement calculation.
What the Forecast Error Calculation and System Should Be Able to Do
One would be able to for example:
- Measure forecast error
- Compare forecast error (For all the forecasts at the company)
- To sort the product location combinations based on which product locations lost or gained forecast accuracy from other forecasts.
- To be able to measure any forecast against the baseline statistical forecast.
- To weigh the forecast error (so progress for the overall product database can be tracked)
Getting to a Better Forecast Error Measurement Capability
A primary reason these things can not be accomplished with the standard forecast error measurements is that they are unnecessarily complicated, and forecasting applications that companies buy are focused on generating forecasts, not on measuring forecast error outside of one product location combination at a time. After observing ineffective and non-comparative forecast error measurements at so many companies, we developed, in part, a purpose-built forecast error application called the Brightwork Explorer to meet these requirements.
Few companies will ever use our Brightwork Explorer or have us use it for them. However, the lessons from the approach followed in requirements development for forecast error measurement are important for anyone who wants to improve forecast accuracy.