How to Best Understand Forecast and Supply Planning Prototyping

Executive Summary

• Forecast prototyping is performing forecast and supply testing.
• Prototyping is critical to understanding how to make changes to the forecasting and supply system.

Companies make a significant mistake when they select one forecasting system and trust that everything in that system will work as advertised. Instead, forecasting should be prototyped in another competitive solution to make sure that the results match.

See our references for this article and related articles at this link.

Pre Reading

It is essential to have a prototyping environment that can allow a company to simulate different forecasting approaches. This includes hierarchical methods, improve forecast accuracy, formulate planning business processes, and test planner workloads and model design trade-offs. This can be done before locking in on a design for a more significant, less flexible system like APO.

How to Understand a Forecast Simulation Environment

The best way to understand a simulation understands how it differs from a production system. While the definition of the differences between prototyping and production design listed below is developed from product design, the description works equally well for software prototyping.

• Materials: Production materials may require manufacturing processes involving higher capital costs than what is practical for forecast simulation. Instead, engineers or prototyping specialists will attempt to substitute materials with properties that simulate the intended final material.
• Processes: Often, expensive and time-consuming unique tooling is required to fabricate a custom design. Prototypes will often compromise by using more flexible processes — lower fidelity. Final production designs often require extensive effort to capture high volume manufacturing detail. Such detail is unwarranted for prototypes as some refinement to the design is to be expected. Usually, prototypes are built using minimal engineering detail as compared to final production intent.

A prototype, therefore, allows the testing and of concepts in a lower cost medium and more flexible. For instance, automobiles are initially designed and reviewed on a computer or in clay before they are fashioned in metal.

When a simulation is built, it may not be manufactured with the final materials used in the production item. However, by creating a reasonable semblance of what the final product will look like, it allows the designers to take input and make changes.

What Makes a Good Simulation Environment?

The secret to good software implementation simulation is to find a suitable simulation environment. The environment must meet the following criteria:

• Inexpensive
• Flexible
• Transparent
• Changes to inputs must be readily observable in outputs
• Strong ability to produce analytics
• Fast

How a Forecast Simulation Environment Compares to a Demo Environment

A forecast simulation is different from a demo in that a demo uses a tiny subset of data to demonstrate functionality. Demos have developed a bad reputation as they have repeatedly been used by software companies to create the illusion of being able to meet customer requirements, often with software that cannot. This is because the person performing the demo, while typically quite knowledgeable in the software, is scripting the demo to highlight only the positives, and to conceal the weaknesses of the application.

On the other hand, a prototype environment may process the same quantity as a production / live system but do so in an offline environment that only the application engineer or analyst can use.

Troubleshooting a Live Forecasting System

I have decided to perform prototyping on a product called Demand Works Smoothie to troubleshoot a problematic SAP DP implementation. This is important because SAP DP is not a good prototyping environment, and the way the system is currently configured is not working correctly. Thus I needed a reliable external system to get the forecasting down to where my client required it.

After reviewing some products, I picked Demand Works Smoothie (formerly known as just “Smoothie”) to do this job because of its strong ability to do best-fit forecasting (called Expert selection in Smoothie). The fact that it has resident within it the forecasting models that I needed to test and the fact that DemandWorks has knowledgeable support. It is essential to point out that DemandWorks can be used for a lot more than I am describing here. However, this is the need that I initially selected Smoothie for.

Data Storage and Manipulation

One of the most critical factors on an excellent prototyping environment is having a fast and easy to manipulate data backend. This is where SAP DP fails. SAP DP’s data backend is so complicated that it takes entirely a lot of time to make changes to it, and in my view, is one of the most impoverished areas of functionality in the entire SAP APO suite. (see details on this topic here.) Data management in Smoothie is straightforward. It read both databases and spreadsheets. Up until this time, I have only connected it to spreadsheets for data uploads.

However, the spreadsheet that is uploaded to Smoothie is straightforward to manage. Mostly, all the different items that are to be uploaded are separate tabs in the spreadsheet. This allows a different model to be saved as a simple spreadsheet. One spreadsheet model can hold anywhere from 2 tabs (if only essential forecasting is performed) to quite a bit more than for different measures, or if supply planning is being done. Smoothie does both demand and supply planning, with the Smoothie SP being an extension or add-on to the Smoothie forecasting application.

The Smoothie manual describes its data management in the following way:

“Almost all data in Smoothie is sorted at the base of the model. The model’s base is the level that the data is imported into. You can easily identify the model’s base level by using the Tree driller and clicking on the plus sign next to the folder.

The dot indicates that you are working with a base level item. Folders indicate that you are working with an aggregated of items (i.e. a hierarchy)

Smoothie does things differently depending upon whether you are at the base level or working with an aggregation. For example forecast model changes made in Visual Forecasting are handled differently depending upon whether you are working with a detail item or an aggregation.” – Smoothie Help

Testing

The steps were pretty straightforward. I needed an extract of my customer’s data per SKU. As my client only forecasted to one location, that decreased the SKU number significantly. Secondly, I decided to remove the deficient demand items from the product database. Insufficient demand was less than three items per year. That reduces the database somewhat.

Thirdly, I need to get only those items for which there were no manual overrides. I wanted to compare a strict statistical forecast in Smoothie to one in SAP DP. If I pulled products which have been manually overwritten, I would have blended the forecasting effectiveness of individual forecasters with the actual statistical method used. I did not want to do that, so I excluded manually altered forecasts. This would now be an apple to apple comparison.

Next, I decided to forecast using Expert mode in Smoothie only, and the system selected the forecasts.

I then exported from Smoothie, and this provided me with a file that showed the method chosen for every product. It also provided me with several different forecast error measurements. The question was whether Smoothie could produce forecasts that were of higher quality than the forecasts produced by DP. This should be relatively easy as the SAP DP implementation that I was evaluating was primarily going out on one model.

Flagging Unforecastable Items

It is an illusion to think that all items in a product database can be forecasted. Items of high variability and intermittency, at some point, cannot be forecasted, and need to be placed on consumption-based logic in supply planning. Thus these product types do not need to be in the planning engine (although they do need to be in the S&OP engine. This can be accomplished by generating two feeds, one from the forecasting engine for forecastable products, and one from the supply planning engine providing non-forecastable products).

In determining which products should not have a forecast, I decided to use MAPE, MASE, and the R-squared (which in Smoothie is a correlation between the statistical forecast and the naive of mean averaged over time.)

In this way, I was able to segment my client’s items into forecastable and unforecastable products. This helped clear out a lot of the product database from the forecasting engine that was not adding any value being there. To find out more about determining forecastability, see this article.

Ease of Doing Things Versus the Ability to do Things: The Problem with Feature and Functionality Comparisons

One of the major misunderstandings about SAP DP specifically (although it applies to SAP generally) is that SAP tends to have a huge amount of functionality. In a “feature and functionality” comparison, SAP tends to win those comparisons. However, the truth is that the software used does not come down to simple features and functionality. This is because an application with a more narrow set of functionality is often superior in usage to an application with a broader range of functionality. After all, the functionality of more narrow applications tends to work better. The straight comparison of features and functionality is problematic as it promotes software vendors to add as much functionality as possible, rather than trying to improve the most critical functionality or to invest in making the functionality unique in some dimension. If vendors think that they will win software selections by cramming not particularly useful functionality into their applications, then that is what they will do. Secondly, the entire feature/functionality “bake-off” orientation pushes vendors into developing capabilities in areas that they lack competence.

If we take the example of SAP DP and restrict the discussion to statistical forecasting, several features make Smoothie far more effective and far more comfortable to use and to maintain than DP.

1. Best Fit Functionality: Smoothie runs the best fit automatically. It runs the best fit procedure as the data is loaded. SAP DP requires the best fit (called auto model selection) to be run interactively (with the user selecting to have it run) or in the background, which must be set up by IT. There are many more details to this topic, but Smoothie runs the best fit procedure far more efficiently and much more rapidly than SAP DP, requiring far fewer technical skills. So while both the applications “have”best-fit functionality, the difference between how the best fit functionality can be run is night and day.
2. Grouping and Hierarchy: When it comes to grouping and hierarchy management, again, the difference is night and day. SAP DP’s complexity concerning hierarchies means that on most DP projects, the hierarchy is rarely changed. Even discussing hierarchies in DP means understanding a specific vocabulary to consider making changes or how they work in the IT sense. Smoothie, on the other hand, can add attributes by simply adding them to a flat file and making the upload to Smoothie. (when using a database this is more involved, but Smoothie can be run with files with no database, SAP DP cannot). With Smoothie, the creation of the hierarchy is performed during the upload. As a user of the software, there is nothing else to do, and the creation of this hierarchy is speedy and very easy to change.

Therefore, the takeaway in this topic is that Smoothie does things much more quickly, smoothly, and with lower effort and maintenance than SAP DP.

The Issue with SAP DP Maintenance

We tune SAP DP systems professionally, and we do not encounter SAP DP systems that are adequately maintained. This may be due to our exposure, but, naturally, systems that are difficult to maintain often will not get the level of maintenance they require. Forecasting departments do not have the bandwidth to optimize challenging to maintain systems and must have an easy to use and manage system if not for forecasting in production at the very least for forecast testing. We would argue that SAP DP has set back forecasting at the companies where it has been implemented. The reason is that keeping DP working technically, combined with the user overhead of only using DP, is so high that it consumes the bandwidth of the planning department, leaving too little left over to focus on the business side of forecasting.

Companies that do not have a system to use other than SAP DP where various hypotheses can be tested demonstrate a strong tendency to use guesswork to determine how to configure SAP DP, and they pay a heavy price in forecast accuracy.

The Logic for Using Both SAP DP and Demand Works Smoothie

Customers that have purchased and implemented SAP DP typically don’t appreciate the concept of removing DP. Secondly, time and money have gone into training resources both in demand planning but also in sales or marketing to use the SAP DP planning book (user interface).

However, it should not be the goal to use as few forecasting systems as possible. There is no reason, no financial reason, no user training reason, and no functionality reason, not to use Smoothie alongside SAP DP. For forecast testing, one of the most underestimated areas of importance in forecasting, Smoothie is one of the best systems available. Companies that buy Smoothie can bring powerful forecasting testing capabilities into their company.

*This should not be interpreted as an endorsement for the idea that forecasting is like “self-service BI” performing this type of testing requires specialized skills and cannot be delegated to people that already have a full plate or significant operational responsibilities.

Smoothie can be used to validate results in SAP DP, and can, to the degree it is accepted by the company, even replace things that SAP DP does. We are focusing on testing in this section, but there are other benefits as well. For example, the company would also bring Smoothie’s excellent user interface into its company, enhancing the ability to provide forecast training to users. Smoothie is also one of the best applications for training people not only on software but on forecasting concepts in general.

After many years working both in SAP DP and Smoothie, we conclude that a combined Smoothie/SAP DP solution would have to have a lower TCO and higher achieved forecast outcome than SAP DP alone. And the measurement should be a combination of the TCO along with the forecasting outcomes, not how few applications the company can get by.

Why Are External Prototyping and Generally Not Used in Supply Planning?

A significant reason that prototype environments are generally not used is that most companies are unaware that they can benefit from them. This information never comes from consultants at the large consulting companies were discussing other systems other than the one to be implemented is considered heretical. Also, there are very few books that described using two environments in which to make comparisons, so even highly paid consultants are often unaware of this benefit. While the benefits of external prototyping for supply planning change depending upon the method employed, all methods can benefit at least somewhat.

MRP and DRP Prototyping

In discussing this topic with an executive at one of my clients. I was told that he had never thought of using any tool to test the results and that they had never used such a tool for ERP.  This may be why tools of this type are not generally used because the model is ERP, and with ERP, the supply chain complexity is sufficiently low that it is often unnecessary. Performing simple MRP and DRP does not generally bring a lot of surprises in the planning results.

The Issue with MRP and DRP Prototyping

This is because MRP is straightforward, and MRP as with DRP is based on straight arithmetic. DRP is a slight modification to the logic of MRP, which I describe in great detail in my book Supply Planning with MRP, DRP, and APS Software. This is why MRP and DRP don’t usually require an external prototype environment.

However, even with ERP (ERP systems typically only use MRP and DRP for supply planning) planning, I can see a benefit to externally modeling MRP and DRP.  Again, because these are more straightforward methods than the planning methods available within APO, the necessity may not be as great, but it is still valuable. For instance, I could model some of what R/3 is doing in an inexpensive prototype environment like Demand Works Smoothie. This application can be turned around and adjusted much more quickly than R/3 or other ERP systems. The master data parameters are elementary to change. They are all kept in a spreadsheet when the application is being run in prototype mode. ERP systems, on the other hand, both have much more difficult master data management. This is because ERP systems have to connect the supply chain portion to all other non-supply chain modules and ERP systems often tend to be “heavy.” However, Demand Works Smoothie allows one to access MRP and DRP in a very lightweight and inexpensive application.

Conclusion

SAP DP has been on the market for close to 20 years and is now a well-known quantity. We conclude that SAP DP cannot achieve the objectives that companies have for forecasting all by itself. This seems like a strong statement, and while it is hypothetically possible, it is not practically possible because no company is willing to staff at the level necessary to get SAP DP to this level of maintenance. Furthermore, doing so would be incredibly wasteful.

The best option for companies that have already invested in SAP DP is to find a second forecasting application, Smoothie being an excellent choice, and use Smoothie to fill in the holes in SAP DP. We have been doing this ourselves with well-tested outcomes.

There is a significant benefit to prototyping both MRP and DRP in a system like Demand Works Smoothie, and it is quite inexpensive to buy the application and inexpensive to maintain and change the master data within the application. However, supply chain departments rarely use external prototyping environments for MRP and DRP even through they could greatly benefit from them. An external prototype environment can be used before go-live, during go-live, and in support. All the time, master data changes can be tested. This is a far more cost-effective method of hypothesis testing versus using an ERP system to make changes within.