What to Look for In a Forecast Simulation Environment

Executive Summary

• Forecast simulation can be used to test important items that allow for the forecasting application to be adjusted.
• Specific items make a forecasting application a good prototyping forecasting.

Introduction to Forecast Simulation

Rarely does the public or do people who work at companies find out about problem implementations. However, I run into a lot of flawed implementations which must be diagnosed. So on several posts, I describe the problems that I have run into. Some of these problems may match issues that others are currently facing.

See our references for this article and related articles at this link.

The Case Study

The basic issue from one project I was working on the statistical forecasting within SAP DP (Demand Planning). DP was not working properly. There were two main issues which were problems.

1. The first issue is that the auto-selection model (which contains best-fit functionality) was not selecting the best model.
2. The second issue is that most the models within DP were not working.

The primary forecasting model that was used was the seasonal model. A related issue is that the MAPE forecast error measurement is not calculating correctly. For more on MAPE in SAP DP sees this article. Many companies are unhappy with the best fit functionality in SAP DP. Details on best fit in SAP DP is covered in detail in this article, so I have already run into this problem several times before working on this client.

The Previous Approach

Previous approaches to this account by previous consultants which attempted to resolve the DP forecasting issue have centered around solving all of the problems in DP. Furthermore, the Deloitte implementation team left the client without ever resolving the issue, and Deloitte had attempted to mislead this client so that they would not have to try to figure out how to solve the issue. Deloitte punted on the topic.

SAP had been contacted about the issue, and both did not resolve the issue but told them things that simply made no sense. Essentially SAP told them that DP was not designed for their forecasting volumes. I found this strange as the volumes were not particularly high. The result is that the problem had persisted for some time.

Our Approach to Forecast Simulation

I very much dislike performing forecast simulation in SAP DP. I describe DP as an oil tanker that takes a good deal of effort to make adjustments within.

Secondly, in addition to getting a more maneuverable system, I also needed a system where I could rely on the forecast results so that I could triangulate them against SAP DP. Therefore my approach was the following:

1. Select a forecast simulation environment
2. Obtain agreement that if the production environment were forecasting in a way consistent with the forecast simulation, the client would be satisfied, and then attempt to recover the DP statistical forecasting by using the prototype as an external check on the DP results.

What is a Simulation?

To develop a shared understanding of the project regarding the approach, it’s necessary to describe what a simulation is. Prototyping is hugely underused as a method for hypothesis testing in systems implementation. And because of this companies can end up spending a great deal of money going down a path that is untested.

How to Understand a Forecast Simulation Environment

The best way to understand a simulation understands how it differs from a production system. While the definition of the differences between prototyping and production design listed below is developed from product design, it the description works equally well for software prototyping.

• Materials: Production materials may require manufacturing processes involving higher capital costs than what is practical for forecast simulation. Instead, engineers or prototyping specialists will attempt to substitute materials with properties that simulate the intended final material.
• Processes: Often expensive and time-consuming unique tooling is required to fabricate a custom design. Prototypes will often compromise by using more flexible processes. Lower fidelity. Final production designs often require extensive effort to capture high volume manufacturing detail. Such detail is unwarranted for prototypes as some refinement to the design is to be expected. Often prototypes are built using very limited engineering detail as compared to final production intent.

A prototype, therefore, allows the testing and of concepts in a lower cost medium and more flexible. For instance, automobiles are initially designed and reviewed on a computer or in clay before they are fashioned in metal.

When a simulation is built, it may not be manufactured with the final materials used in the production item. However, by creating a reasonable semblance of what the final product will look like, it allows the designers to take input and make changes.

What Makes a Good Simulation Environment?

The secret to good software implementation simulation is to find a good simulation environment. The environment must meet the following criteria:

• Inexpensive
• Flexible
• Transparent
• Changes to inputs must be easily observable in outputs
• Strong ability to produce analytics
• Fast

SAP SCM DP meets none of these requirements. This makes DP a very expensive and ineffective environment to use for forecast simulation.

How a Forecast Simulation Environment Compares to a Demo Environment

A forecast simulation is different from a demo in that a demo uses a very small subset of data to simply demonstrate functionality. Demos have developed a bad reputation as they have repeatedly been used by software companies to create the illusion of being able to meet customer requirements often with software that cannot. This is because the person performing the demo, while typically quite knowledgeable in the software, is scripting the demo to highlight only the positives, and to conceal the weaknesses of the application.

On the other hand, a prototype environment may process the same quantity as a production / live system but do so in an offline environment that only the application engineer or analyst can use.

Software Selected

After reviewing a number of packages including SPSS and EZForecaster and John Galt, I selected Demand Works Smoothie, which you can read about in this article.

Conclusion

The project resulted in the determination that the wrong best fit auto-selection was being used. Only one of the two auto-selections in DP works. A second outcome of the project was that one reason that the client thought DP was malfunctioning was that the best fit would often select a level forecast.

However, the client had very lumpy and unforecastable demand. When this occurs, the best fit selection in DP, or any other forecasting system, will select a level forecast. Therefore, this was not a flaw in DP. However, it brings up the discussion of whether these types of products should even be forecasted.

Interestingly, after Smoothie was used as a forecast simulation environment, it proved very useful because of its highly usable user interface, and its innovative attribute-based forecasting capability.

This project resulting in me using Smoothie very frequently and for it becoming my preferred forecast simulation environment. I also developed a model for using Smoothie for all manual adjustments and the importing the adjusted data into DP. In summation, the prototype approach, and the selection of Smoothie not only helped this project but resulted in the discovery of a tool which has been highly useful for other projects that face similar issues.