How to Understand Forrester's Fake S/4HANA TCO Study

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

Forrester created another TCO/ROI study that was paid for by SAP.
We review the accuracy of this Forrester SAP S4HANA study.

Introduction

Forrester has a history of taking money from SAP and producing precisely the outcome results that SAP would have wanted. Forrester will do this for any vendor and frequently does, as it is a significant part of their business model. This study by Forrester is an entirely fake TCO study designed to be used by SAP’s marketing and sales to mislead its prospects into thinking that S/4HANA has a low TCO. The study is hilarious and shows how the paid-off research system works. You will see how Forrester concocted this faux study with its analysis of S/4HANA.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: We have no financial ties to SAP or any other entity mentioned in this article.

This is published by a research entity, not some lowbrow entity that is part of the SAP ecosystem.
Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. As you are reading this article, consider how rare this is. The vast majority of information on the Internet on SAP is provided by SAP, which is filled with false claims and sleazy consulting companies and SAP consultants who will tell any lie for personal benefit. Furthermore, SAP pays off all IT analysts -- who have the same concern for accuracy as SAP. Not one of these entities will disclose their pro-SAP financial bias to their readers.

Forrester Gets to Talk to Three Companies?

According to Bill McDermott, S/4HANA is the fastest-growing application in the history of SAP. So why did SAP only make three customers available to Forrester? It does not take much time to interview customers; as SAP was paying for the study, one would think they would try to bring more customers to the table.

Forrester found the following financial benefits.

Three Year Benefits

These look interesting until you see how these numbers were determined. The logic for these numbers does not make sense, as will be seen from the quotations. Forrester did not find any of these cost savings, which are all fake.

Quotations from the Study

Let us look through some of the impossible quotes from the article. These quotes were stated to Forrester by executives in the companies provided to Forrester by SAP, and Forrester did not bat an eye.

A potential reason for this is that a) Forrester is being paid here to look the other way, and b) it’s scarce for anyone who works at Forrester to have implementation experience.

But to a person with implementation experience, these quotes make little sense.

Complex Supply Chains Lead to Lack of Inventory Visibility?

“Our supply chain from extraction to market is exceedingly complicated. We have over 157 different avenues for how we go to market. By the numbers, we have 60,000 unique materials, 6,000 vendors, 4,000 customers, and an additional 10,000 partners.”

S/4HANA’s supply chain functionality still lags ECC by a substantial margin. Secondly, a complex supply chain should not impact inventory management. No matter how many, each of these locations is a simple setup as a plant material combination connected to another plant material combination. I have personally had many customers with the numbers listed above, so it is not surprising or more complicated than what many other companies deal with and use ECC. It is a simple goods issue and goods receipt in ECC. If this quotation were from 5 years ago, ECC would be considered what solved this issue. Pre-S/4HANA, if you were to have asked SAP if they could handle..

“157 different avenues for how we go to market. By the numbers, we have 60,000 unique materials, 6,000 vendors, 4,000 customers, and an additional 10,000 partners.”

Their answer would have been that ECC is running the supply chains of the largest companies in the world, so the scenario above is nothing they have not done. The study is not precise that the company moved from ECC; they may have moved from another ERP system. However, this study is trying to move the burning platform to S/4HANA when almost all the vast supply chains using SAP worldwide are running ECC.

S/4HANA Provides Real-Time Inventories?

“The most important change is that we now know the real inventories at our mine locations and operating plants. This gives us visibility from the extraction point to the final shipping to buyers. We can now measure real-time inventory across the entire company.”

All ERP systems have immediate visibility to inventory and have had this for many years. The term real-time inventory is used. However, systems stopped advertising “perpetual inventory” many years ago, as all systems that deal with inventories have this. That is, it ceased to be a differentiator, where at one time it was. Perpetual or real-time inventory was one of the original selling points of purchasing MRP systems in the 1980s. We cover this in the article Whatever Happened to the Perpetual Inventory.

Note to executives: please stop saying things that reflect information technology crazy mouth, like your new system provides real-time inventory. Part of the logic for why you are paid so much more than employees is based on the concept that you know more than the average person. When you say things like this, it ruins the illusion.

SAP and compliant customer executives have a habit of proposing that a new item newly solves a problem that was already supposed to have been solved with a previous version. If automobile manufacturers followed SAP’s lead, automobile ads might look something like the above.

Sales People No Longer Losing Deals Due to Faulty ATP Checks?

“Our salespeople were losing deals because we could not promise when the products would be delivered. We just didn’t have visibility into our supply chain to be able to make commitments when our competitors could and did make promises. A seemingly small system problem became a competitive disadvantage.”

It isn’t easy to see how the situation with S/4HANA will be much different. S/4HANA does not have superior ATP functionality over ECC. Is this company having a problem with ATP because of functionality or other reasons, like their production schedule changes too often? There can be many reasons why the available quantity can be incorrect. However, this is too much nuance, and Forrester has a preconceived conclusion they need to reach, so let’s move on.

All Companies Benefited from Reduced Operating Costs and Inventory Carrying Costs?

“Each of the companies that Forrester interviewed reported significant improvements in managing operating costs, such as inventory carrying costs. Using SAP S/4HANA enabled the composite organization to combine cost data with inventory data in a way that enabled employees to optimize flow and cost of inventory.”

This might be a good time for Forrester to re-emphasize that SAP did not allow Forrester to speak to any customers who bombed their S/4HANA implementations or even to the average S/4HANA customer. However, our research supports that most S/4HANA implementations (as covered in the Implementation Study of S/4HANA) have been unsuccessful. That is, the failure rate is far higher than 50%. Both SAP and the giant SAP consulting firms currently cover this failure rate. SAP also uses an NDA for its customers that implement S/4HANA, which means that no matter how bad the experience, customers can’t talk about it. Forrester did not see fit to report on the S/4HANA NDA.

This is the problem with Forrester posing as researching while its entire sample is unrepresentative of the population. What is the value of a wholly rigged sample designed to lead to a predetermined outcome? The answer is zero…or, perhaps, negative.

The Company Reduced the Need for 600 Employees…And No One Was Fired?

“The financial model is based on a composite organization that was able to reduce the need for 600 employees. Although these staff will produce additional revenue in the future, the immediate benefit is measured by the reduction in employees required to do the same work. The future value is discussed in the Flexibility section.”

This is equivalent to having your cake and eating it too.

Cost Savings…..But With No Cost Savings!

It is doubtful that a company would make 600 employees redundant but keep them on staff. Companies today are too bottom-line-oriented. Is every one of these people an excellent fit to be moved to a new job?

Hmmmmmm…… Some of those employees, if not most, would have been fired. However, a company can claim cost savings without admitting they fired people. This political language circumvents the discussion of what happens to employees supposedly reduced by systems. However, this is also quite overstated. None of the projections around the labor savings of ERP systems turned out to be exact; what did happen was companies were required to staff up IT.

Notice the following quotation as to the financial benefit of the moved employees.

Massive Revenue Unrealized Because of Lack of Employees

“By centralizing planning, production, and tracking functions, we eliminated the need for 600 employees. We redeployed those people into new locations to open new mines and refineries. It’s still too early to claim any revenue, but within a couple of years, we expect that these same employees will drive growth that will be hundreds of millions of dollars.”

Let us assume a future revenue of $250 million (they don’t specify how many hundreds of millions, only plural, so $250 million would be conservative). Divided by 600 employees, this is $416,000 per employee. If this type of opportunity were available to the company, they now look incompetent to have not hired them before the S/4HANA implementation freed up 600 people. Let us review what we know about this mining company.

They don’t understand that their inventory system has been real-time for decades.
They allow investment opportunities to wither on the vine instead of hiring people to drive hundreds of millions in revenue.

How badly run is this mining company?

S/4HANA Took Nine Months to Implement?

“The composite organization no longer needed to pay the annual subscription costs or maintenance fees for older applications. The company executives told Forrester that it took an average of nine months to implement SAP S/4HANA.”

Naturally, the implementation times reported are far less than the implementation for ECC, a one-year minimum, but often implementations stretch to 2 or 3 years or more. This is true even though S/4HANA is far less mature than ECC, and immature products take longer to implement than mature products. Every single SAP-reported implementation duration is shockingly fast.

However, when information that SAP does not control is reported, the timeline is quite long. In the case of the Lidl failure, it was a total of 7 years, with S/4HANA being a subcomponent of that as they transitioned to S/4HANA after beginning with ECC. Haribo, another failure, was also seven years old. Yet the companies interviewed by Forrester averaged nine months. If we include Lidl and Haribo’s implementation duration, what happens to the average of Forrester’s sample? Forrester did not even have to interview these companies as the project’s duration and the cost were published due to the project’s failure.

Real Implementation Data Points Keep Rolling In that Contradict Forrester

It was later learned that the Schweizer Post S/4HANA implementation rose to $84 million.

Wait a second; why did this implementation cost even more than $1 million? According to Forrester, a typical S/4HANA implementation is around $877,000 and runs around nine months, with only two consultants who stay on the project for six months. Forrester’s study of SAP-provided sources showed almost no variance in this.

Now, hearing of an $83 million implementation is very distressing. According to Forrester, the 12X cost overrun seems odd, as this amounts to $83M/.87M, which is 95X the typical cost of a S/4HANA implementation. This cost will rise in the future. If we include just this implementation into the Forrester “study,” we end up with (3 * $.877M + $84M) or $21M. Could there be some reason that the Schweizer Post case study was not provided to Forrester by SAP? Isn’t SAP interested in providing the full context of data points to Forrester? It’s a study, right???

It is curious. We keep teaching statistics and talking about the importance of math, but we don’t seem to have research entities where you can use this math honestly. Forrester can publish a study with three rigged case studies, and it goes with little notice. Math + lying does not lead anywhere.

Sales Orders are Entered in Manually?

“We design complex systems for our customers, but then the information must be converted into orders and placed into SAP. Today this is a manual process, but we are looking into the idea of having the information convert directly over, which will reduce delays and avoid errors.”

This company has not yet figured out how to upload externally generated sales orders to S/4HANA. Interesting.

These implementation costs are ridiculous. ECC and S/4HANA implementations take many months of consulting support. In this case, the consultants rolled off three months before the go-live (or they were part-time).

Immature ERP System Implemented with Two “Swiss Army Knife” Consultants

S/4HANA is a risky implementation because so much of it is still immature. If only two consultants were used, how would that have worked? A minimum is to have a consultant for materials, finance/accounting, and sales. (it is unclear if production was implemented).

Which of the two consultants covered two areas?

And further, who covered HANA? S/4HANA must be implemented with HANA, a high-overhead database. S/4HANA consultants are not database consultants. So, did one consultant implement S/4HANA, and the other consultant implemented HANA?

It also means there could not have been any variability in the number of consultants. Unless one project had one S/4HANA/HANA consultant and another had three.

Does this mean that one consultant implemented HANA while the other implemented S/4HANA? Or HANA took three months of consulting time to implement, and S/4HANA takes 12 months of consulting time (summing to 9 * 2 = 18 total months of consulting)(that is, for some of the time, there were 2 S/4HANA consultants and sometimes not?)
So those two consultants are “Swiss Armied” to an extreme degree. Could they run this lean on three different projects with three different sets of consultants?

Unrealistic Implementation Durations…..from Forrester

Every report by Forrester highly understates the implementation costs to customers. Some clients have 70 to 100 SAP consultants on-site (for multiple products), and with immense monthly billings, every client Forrester seems to interview has a tiny amount of billings. However, they will laugh at you if you approach Deloitte and ask them to help you implement S/4HANA with two consultants for six months.

Our Golden Pinocchio Award

For this claim, Forrester wins our highly coveted Golden Pinocchio Award. However, as SAP funded the lie, and SAP knowingly provided erroneous information and would have manipulated the customers into providing false information, the award is co-shared with SAP.

This estimate is even less believable because it is an average, which means that these minimum values were the three companies’ average. That is, this is not from a single company that had some minimal scope.

We can say with high confidence that this did not happen.

Either the three companies were instructed to lie to Forrester, or Forrester changed these values as directed by SAP.

Inverted Implementation Versus License Ratio?

Notice the license/subscription costs that are a significant part of the costs for S/4HANA. Something that should be a major red flag is that the license costs are far below the implementation costs.

In fact, the ratio of implementation costs to license costs is $833,750 / $3,000,000 or 27%. That is never the case; the implementation expense on SAP projects is a multiple of the license cost, not a fraction of the revenue. Even an SAP sales rep with a few months under the belt knows this (it comes up during discussions around implementation estimates for customers). And the fraction here is far less than 1. If a multiple of 2 were used, that would mean that all three customers in the study 2/.27 = .135 or 13.5% of a typical implementation. Two is the number many consulting companies use in the sales process, but it is far too low. Readers can see for themselves at the Brightwork ECC TCO Calculator (this is for less than 800 users; see the included link for more than 800 users. But even a more straightforward ECC implementation for 500 users will run up more than $20 million in consulting fees.

However, SAP makes most of its money from licenses and support, so the unrealistic consulting costs are not SAP’s problem; they are Deloitte and Accenture, etc… ’s expectations to manage.

Forrester is Still Learning About Implementation Versus License Revenues

Is Forrester seriously unaware of the implementation/license multiple demonstrated through decades of SAP implementation experience? Let us look at the Lidl example again, which was 500 million euros. ECC implementations are routinely $100 million and often much more. SAP even has multi-billion dollar failures on its resume. The Oracle ERP “legacy replacement” (it is well known that Oracle ERP has less implementation costs), with the Air Force, cost $500 million before the program was stopped. Sub-million dollar ERP implementations from commercial vendors generally do not exist, particularly with SAP. Forrester seems unconcerned that other people will also read this study and see how impossibly inaccurate it is.

And the errors get perhaps (even more?) or just as extreme with the following calculation for license costs.

Where are the Costs for HANA?

Forrester is quite unclear as to what version of S/4HANA was implemented. SAP is utterly misleading about how different the S/4HANA Cloud is from the on-premises version. S/4HANA Cloud can be implemented quickly, but its scope is minimal, and given the licenses, which show a $3,000,000 on-premises charge, these were on-premises S/4HANA versions.

And this brings up a related question. Where is the license charge for HANA? HANA is a costly database, and S/4HANA can only run on HANA, so Forrester apparently left out the cost for HANA.

These previous two points must be taken together. Not only is the implementation versus license multiple wholly deflated, but the license numbers are missing the most expensive component of the overall BOM, the HANA database.

Forrester has no fear of making errors (or deliberate underestimations), knowing that some people will read the entire study.

Analyzing The Vendor or Vendor Funded TCO Estimates

Vendors view TCO discussions within the context of making a software sale. Some vendors still produce TCO calculations, and we have analyzed a number of these TCO calculations. We have used some of these estimations for our estimations and have found some good and some bad in how they were put together.

Several of the most apparent issues are listed below:

No Complete or Full TCO

I have yet to come across a true “TCO” produced by a vendor or any vendor-sponsored study (which I view as the same).

While a few have been accurate in several respects, they did not estimate the client’s cost of implementing or maintaining the application. However, the TCO is that; it must include the costs to bring the application into the company, keep it running, and decommission the application. I have certainly appreciated the work put in by software vendors to estimate some of the costs, but it is confusing to see “TCO” in a publication’s title and then read the following.

“…does—NOT—include analyst costs, IT costs, local data management costs, etc.”

One could imagine the following riddle:

“When is a TCO not a TCO? When it does not include all costs.”

Incorrect Duration Estimation

Vendors have a strong tendency to underestimate the duration of the software implementation. For example, in the published S/4HANA case studies, the numbers published by SAP were far shorter than the known period of ECC implementations, even though ECC is far more mature than S/4HANA. Therefore, part of underestimating TCO is underestimating the time it takes to go live.

It may undoubtedly be true that the application could be installed as quickly as estimated in a laboratory environment, but companies are not laboratories. Some vendors (other than those like SAP and Oracle, both of which have business models of lengthy implementations due to the consulting companies’ need to outsource their consulting), will complain that the client is what holds up the implementation. This is a bit like a doctor complaining that if only patients had medical degrees, they could see many more patients and get through the day faster. It’s simply not relevant to the question of duration estimation. The following statement is included in the TCO questionnaire that we give to vendors:

“Please use historical timelines rather than ‘optimal scenario’ timelines.”

Therefore, the client’s knowledge level and the time required for knowledge transfer must be estimated, along with all the other project realities. Another example of this is integration. IT departments are slow to respond with extract creation.

Software Vendors Faking Enthusiasm for TCO

While most software vendors hide their pricing information, all software vendors still like to talk about TCO in the abstract. Being in favor of TCO is like saying you favor the American flag or apple pie; you favor goodness. You can’t find anyone who opposes TCO in principle. But the devil is in the details. As long as nothing is quantified or the TCO studies are rigged, all vendors can say they have the lowest TCO. Most software vendors are against publishing TCO estimations and are even against publishing pricing information for their applications on their websites.

It is certainly not difficult to find examples of software vendors creating false financial estimates for their clients to promote sales. I used to work for one of these companies and witnessed this firsthand (although I am happy to say I never participated in or supported this). The software vendor i2 Technologies offered a Strategic Opportunity Assessment (SOW), which estimated the financial implications of using its software.

“Another company, whose i2 deployment was two years behind schedule because of problems with i2 software, told Nucleus the following I2 did an ROI calculation for us during the sales process it was very positive. But based upon our experience now, we will [independently] evaluate ROI in the future.” – Nucleus Research

How any company could believe a TCO or ROI study developed by a software vendor and part of a vendor’s sales process is, well, strange. In a surreal turn, this company also priced its software based on this value estimation. Talk about a conflict of interest! The following quote from Nucleus Research is educational:

“One customer, that is still cited on the i2 Web site (2003) as a success story but stopped using i2 more than a year ago, told Nucleus:

‘What killed the ROI was the fact that i2 priced their modules based on an ROI calculation that they themselves estimated during the sales process. (i2 stated) that this would lead to $40 million in benefits. (emphasis added) There are definitely benefis to the solution, but their pricing is too high. Sometime, I should take the time to contact i2 and say you really need to look at your pricing.’”

I worked with people from this group at i2 called “Strategic Services.” Strategic Services would falsify any data to obtain a sale. Strategic Services was filled with MBAs from the top schools. They were not technologists and did not know the software except superficially. Their job was to perform financial estimation for prospects that would lead them to purchase i2 applications. This was called an SOA or strategic opportunity assessment.

This is the value of TCO or ROI estimations from vendors or vendor-sponsored studies — zero or below zero if you include the damage of following the advice from such studies.

The Fictional Nature of ROI Estimates

When developing the estimation approach, I never even attempted to create an ROI estimator, as it never seemed to be a realistic goal. However, even if ROI could be calculated, it is doubtful that any software vendor or consulting company should ever be entrusted to perform the calculation. After all, they are attempting to justify software purchases, so they are most obviously a biased party. And this was precisely how SOAs worked at i2. Having seen several SOAs and worked on an SOA engagement providing technical support, I found that they produced SOAs that were utterly unrealistic. The SOA would propose massive cost savings with very little to support the cost savings.

Value-Added Selling

Things reached an extraordinary place at i2 when Strategic Services began pricing some deals not based on a typically negotiated software license fee but on a percentage of the cost savings that were to come from the software. As if i2 did not have enough bias in providing ROI estimates for its software, it further increased the bias by making sales based upon a percentage of the ROI estimate. This was an entirely fictional world in which Strategic Services lived, and for whatever reason, prospects often bought the unrealistic ROI estimations produced by Strategic Services.

This was an entirely fictional world in which Strategic Services lived, and for whatever reason, prospects often bought the unrealistic ROI estimations produced by Strategic Services.

The Impact of i2’s SOAs

The impact of the SOAs was to reduce the quality of decision-making among many buyers. It convinced many buyers that their purchase would be essentially free as they would receive such an enormous payback on the software. But it was all a mirage.

Go to top

Conclusion

This TCO/ROI study gets off on a lousy footing by having SAP control the sample entirely. The sample is rather tiny (3 companies) for what is supposed to be a product that many companies have implemented (according to SAP). SAP was unwilling to bring many customers to the table, which, of course, is a major red flag.

Forrester’s study gets worse with nonsensical quotations and then ultimately falls apart, showing its seams with the ridiculous average implementation time and the costs that will never be experienced by any company implementing on-premises S/4HANA. And Forrester tops it off by excluding the cost of HANA, which would have significantly added to the cost for each of the three customers that Forrester spoke to.

This study allows SAP and Forrester to tell readers,

“We think you are fools.”

For vendors, the study illustrates that Forrester is “open for business.” If you want to say your applications have an infinite ROI or can be implemented without a database, Forrester is your go-to source to falsify a TCO/ROI study.

Something else to note: while Forrester was paid to make this fake study, Brightwork was not paid to analyze it and show how false it was. No vendor or consulting firm will pay another entity to fact-check their statements and ridicule them publicly. Following a profit-maximizing model, one would never write an article like this. This is why most information providers operating in enterprise software don’t do it. And it is why Forrester is so comfortable writing such a blatantly false article. Any experienced SAP consultant who reads this will see it is fake, but they will stay quiet as a mouse on LinkedIn. Most of them work for organizations that also publish false information, and the only information they are allowed to release is information that is promotional of SAP.

And it is not only Forrester that is open for business. ComputerWeekly, Forbes, etc…; the enterprise software market operates so that the entities that provide the most inaccurate information receive the bulk of the income available.