How Accurate Was Forrester’s Study on the ROI of SuccessFactors?
Executive Summary
- Forrester was paid by SAP to produce a report that showed SuccessFactors had a high ROI.
- Numerous errors in the report demonstrate that the study was rigged to provide SAP’s intended outcome.
Video Introduction: How Accurate Was Forrester’s Study on the ROI of SuccessFactors?
Text Introduction (Skip if You Watched the Video)
Forrester published a study named The Total Economic Impact Of SAP SuccessFactors HCM Suite Cost Savings And Business Benefits Enabled By SAP SuccessFactors HCM Solutions. This was a plaid placement by SAP where Forrester worked off information and a minimal number of SAP samples. Forrester did not question if any of the information was true or representative of the reality of SuccessFactors HCM. The study was designed to show that SAP SuccessFactors HCM suite had a substantial ROI. You will learn about our review of the study and the score we applied for its accuracy.
Our References for This Article
If you want to see our references for this article and other related Brightwork articles, see this link.
Notice of Lack of Financial Bias: We have no financial ties to SAP or any other entity mentioned in this article.
Forrester’s Quantified Financial Benefits
Forrester declares the following financial benefits from a sample of customers provided to Forrester by SAP.
“The composite Organization experienced the risk-adjusted present value (PV) quantified benefits of the following services totaling $11,005,505 over a three-year period (see the Financial Analysis section for more details):
Employee Central and Payroll: $1,189,136.
Recruiting and Onboarding: $2,921,945.
Performance, Goals, and Compensation: $6,194,531.
Learning, Succession, and Development: $702,893.”
Costs
Forrester declares the following costs from a sample of customers provided to Forrester by SAP.
“The Organization experienced the following costs totaling $5,906,920 with a risk-adjusted present value of $5,186,681 (see the Financial Analysis section for more details):
Internal labor to plan and deploy SAP SuccessFactors: $60,000.
Incremental hardware, database and operating system license, and maintenance: $0.* The Organization incurred none of these costs with the SAP SuccessFactors cloud solution.
SAP SuccessFactors (or partner) fees for professional services implementation assistance: $265,000.
SAP SuccessFactors subscription fees: $3,751,600.
SAP Preferred Success fees: $750,320.”
How Many Customers Were Used for the Sample?
Forrester stated the following concerning the sample that is used in the study:
“For this study, Forrester conducted interviews with six SAP SuccessFactors HCM customers. Interviewed customers are described as follows (each requesting anonymity)”
However, when one reviewed the list, there were not six entities listed. There were seven. A company each in the following industries:
- Manufacturing
- Retail
- Local Government
- Insurance
- Transportation
- Mining
- Outsourcing Provider.
There is no other interpretation than Forrester miscounted the companies that were in the study. While that seems like a major error, it is, in fact, only a 14% error, but it just looks worse than it is because it is so apparent. As we will see, that is a high point in terms of accuracy for the study.
Sample Selection Bias
The second thing we noticed was the undeniable selection bias of the sample used by Forrester.
One of the foundational quality checks in research is determining if the sample is reflective of the population. Anyone who has taken statistics would know that this subject alone makes up a significant course component. Before any descriptive statistics are used on a sample, the first question must be answered as to representativeness. It is like a chain; the chain’s primary links have to be unbroken for the pulling on the rest of the chain to work.
Getting 30 Samples?
The rule of thumb (called the Central Limit Theorem) requires 30 samples or that are randomly selected from a population to be relatively confident that the example is reflective. This rule of thumb is memorable to most people who have studied statistics, and the following limitations apply.
- This is the minimum sample size, not the optimum sample size. As explained by Christopher Rout, “It’s not that “30 in a sample group should be enough” for a study. It’s that you need at least 30 before you can reasonably expect an analysis based upon the normal distribution (i.e., z test) to be valid. That is, it represents a threshold above which the sample size is no longer considered “small.“
- Many researchers consider 30 the minimum number of samples to conduct a test. When we performed our research into S/4HANA Implementations, we had more than 30 samples. However, in many cases, these many samples are not available. But in this case, as the SuccessFactors team had access to so many customers, one has to wonder why such a small number of samples were used. SuccessFactors is not like some SAP products where it is challenging to find pleased customers. But the budget available to perform the study cannot be underemphasized. Neither Forrester nor SAP has any research orientation. SAP is trying to get marketing collateral at the lowest possible cost, and Forrester wants to make some money. This would have played a role in why so few customers were selected.
- The confidence interval is a statistical measurement for how likely it is that a sample represents the population from which it is drawn. The confidence interval is a measure of probability.
However, we are getting ahead of ourselves because statistics can only be applied if the sample is random. If the sample is cherry-picked, then no statistics of any type can be applied. When cherry picked, the probability is 100% that the example is not representative of the population.
This is sort of the amusing thing about statistics. One can take many statistics courses, but for most of life and analysis, just the basic statistics are necessary to understand because, in most cases, people are not even applying basic statistical standards when evaluating information.
Now let us address why this sample is so biased.
What Customers Was SAP Likely to Provide to Forrester?
Right off the bat, as each of these companies was provided to Forrester from SAP to interview, a question of selection bias immediately comes to mind. The first question is, “why these six companies?” versus what must be over a thousand customers that SuccessFactors has.
- Most likely, SAP would have provided the six most successful implementations convenient for SAP to find and with which they had an excellent relationship.
This is not a random sample; this is an exceptional sample of SAP Success Factors customers. This means that companies that attempted a Success Factors implementation and failed were excluded. Average Success Factor implementations were excluded.
Yet this is nowhere in the disclosure on the first page. Under disclosures, it does say the following:
“The study was commissioned by SAP SuccessFactors and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.”
There are no other study applications, so it isn’t easy to see what this paragraph means.
Further, it states.
“Forrester makes no assumptions as to the potential ROI that other organizations will receive.”
If the study’s ROI estimate is not usable by those that read the study, what was the point of publishing the research? Moreover, the disclosure is right because the sample has been intensively cherry picked, the ROI is not usable — and if the math were correct, which will be explained later in the article, it isn’t. And finally, when SAP publicized the study, they contradict Forrester’s disclosure. They gave readers the impression that the study is reflective of a…
“a typical organization”
Now let us review how Forrester calculated (or represented the calculations from customers) the SuccessFactors implementation’s financial benefits.
The Financial Benefits Estimates
The estimates of financial benefits are broken into the following categories.
- Employee Central And Payroll Benefits: Calculation Table is based upon an internal staff cost of $90,000 fully loaded.
- Recruiting And Onboarding Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour.
- Recruiting And Onboarding Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour.
- Performance, Goals, And Compensation Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour and managers’ cost of $55 per hour.
- Learning, Succession, And Development Benefits: Calculation Table is based upon a fully loaded internal labor cost of $90,000 per year.
The savings in the study came from the reduction in labor costs.
These aren’t estimates; they are what the companies said happened. If this is true, then each of these companies would have had to have eliminated HR positions. One commentary might be that the positions were not removed, but they were merely “reallocated to other parts of the company.” Companies sometimes write things like this as they don’t like it having it pointed out that cost savings often result in resources being fired. But what if all of the HR employees can’t be fired? Well, that would impact the ROI.
The Implementation Costs
The seven sample companies’ implementation average is stated as $265,000 for consulting and $60,000 for internal resources over three years.
The problem here is there is no way this is true.
Jarret Pazahanick, an SAP and SuccessFactors consultant who has done 50 SAP and SuccessFactors implementations, estimated that both the internal cost and the consulting cost would have been “5-10-15X.” This is modified to be 5x for the internal resources and between 12x to 15x for the consulting resources. Jarret’s estimate is far higher in credibility than any SAP estimate because Jarret is not being paid by SAP and is not trying to sell SuccessFactors but works as an independent consultant. I face the same issue. As a long time SAP consultant, my estimates are always longer and more expensive than those produced by people with a vested interest in selling software. When I made these estimates, salespeople who had never worked on an SAP project would tell me they were confident my estimates were too long and that they would interfere with the sale.
Jarret went on to state the following in a LinkedIn comment.
“It is SO beyond the realm of what a real implementation of SuccessFactors really take it is mind boggling. The backchannels are already start to buzz of upset partners and customers wondering why their implementations and quotes are so much more that what SuccessFactors provided Forrester as what the cost would be to get that great ROI (Obviously all made up).”
If the middle figure is taken from Jarret’s estimate, we end up 5x for the internal resources, and let us take a midpoint of between 12x and 15x for the consulting resources. This is 13.5x. 13x of 265,000 is $3,577,500 for consulting. 5x of 60,000 is 300,000.
If we redo the ROI using these estimates, we end up with the following.
The Overall ROI
Now we can contrast this with Forrester’s analysis of the ROI.
“Based on Forrester’s experience, the six-month payback period and 112% ROI numbers are strong when compared to other enterprise cloud implementations. In addition, the interviewed customers that migrated from SAP HCM on-premises software to the SAP SuccessFactors cloud solution had better benefit results.”
So we calculate 31.34% ROI. Even that would be a very high ROI for any IT implementation. However, the sample is cherry-picked, so that ROI is meaningless. Furthermore, there are several problematic features of the benefits estimation.
- There is no independent verification being performed by Forrester. Forrester asked the customers themselves to perform the estimates.
- Customers have a bias to estimate their benefits higher and their costs lower as those that are queried were often part of the software selection. Therefore the responses they provide are a way of congratulating themselves on their decision-making skills.
- Customers, in my experience, don’t typically know these things. This is because they focus on keeping fewer employees capable of performing any research or estimation. Now, there is nothing to stop companies from hiring these skills. However, they usually don’t, as they don’t see them as pertinent for their objectives.
Therefore, even among the cherry-picked sample, the estimates are most likely extremely unreliable. As we just analyzed, the estimates of the internal and external costs are off by order of magnitude of somewhere around 10x, and the costs are the easiest areas to estimate! This is because you have receipts for what you paid for something. The problem is that Forrester received the consulting fees from SuccessFactors and would not themselves know the cost of a SuccessFactors implementation.
Benefits are far more tricky to estimate. This is why at Brightwork Research & Analysis, we have online TCO calculators. Unlike Forrester, no one paid for these calculators to be produced, so they show a far higher TCO than any of the vendors would like to see published.
However, we did not attempt to estimate the ROI of the 53 different applications we estimated. By way of example, review our analysis of the problems faced with calculating lost sales and forecast error costs to appreciate the difficulty of estimating benefits from software implementations.
Curious Questions
Secondly, how did all seven companies end up measuring the benefits in the same way (that is, with labor savings)? The answer is that there is no way they did. This means that the companies reported back on a survey to Forrester. Forrester set up the construct of the cost savings, and the cherry-picked customers responded to the study. This increases the “fishiness” of the study because it means that even less thought was put into the estimates. For example, did every one of the customers agree with the benefit calculation approach? That would be odd if that were the case.
Conclusion
The study is interesting in that it illuminates Forrester’s process. However, the study’s output is not useful for anything for those seeking to evaluate if SuccessFactors is a good investment. SAP paid Forrester to produce a study showing that SF has a very high ROI. And Forrester produced the study that said what SAP wanted it to say. Observe how SAP marketing explained the study.
“SAP SE (NYSE: SAP) today announced the results of Forrester Consulting’s “Total Economic Impact™ of SAP SuccessFactors HCM Suite” study,* which found that a typical organization** that invests in the SAP SuccessFactors HCM Suite could realize potentially $11 million in total quantified benefits over a three-year period.”
Is that what the study found? No.
Also, are seven companies selected by SAP reflective of a typical SF implementation? That is as close as one can say that the study was reflective of what an average company should expect, which is the opposite of what the disclaimer on the Forrester study states. SAP has a history of misrepresenting the findings of a study they pay to have performed. SAP has no consideration for the reputation of the entities they buy research from. Their perspective is the same as what they apply to customers, which is to extract the most from every possible transaction.
Also, notice the endorsement from the President of SuccessFactors.
“In our view, Forrester’s holistic study shows that SAP SuccessFactors solutions can drive not only better employee experiences but also significant cost savings,” said SAP SuccessFactors President Greg Tomb. “We are proud to be helping businesses better understand and serve their employees while simultaneously streamlining business processes and reaping significant cost benefits.”
From his history at Accenture and SuccessFactors, Greg would have known immediately upon reading the study that it was erroneous, but he says nothing but positive things.
Something else incredible is that Forrester declares in the disclosure that they retained “full editorial control.” This seems to imply that SAP had no control over the content produced. Let us say for the sake of argument that the study was performed, and the result was that SuccessFactors had a negative ROI. Would that study have been published? If a study can only find a positive outcome, then it is not a study. It is a marketing document. SAP could have produced the same “study” themselves, but they knew no one would buy the conclusions. So they hired Forrester to rig a study for them.
There is no way out of this. A report like that can’t get released in error. There are a bunch of different people working on it. And they can’t say, “it got messed up when sent to the printer.” If you need “quality assurance,” what is going on over there, I perform that type of research. And while I have typos, you don’t end up with a crazy number like that because you would see it and correct it.
But what is funny is that while they manufactured the numbers to get the high ROI, the consequence is the implementation cost was ridiculous. So now partners are like
“Hey we are charging way way more than that, and our clients are going to come back on us!”
That is what happens when you are dishonest and backward engineer numbers. SAP is probably not happy either. SAP wants a rigged analysis, but they want one that looks credible. Forrester did this with their HANA analysis years ago, which we covered in the article How Accurate Was Forrester on HANA TCO?. But it went under the radar.
Accuracy Rating
The study receives a 1 out of 10 for accuracy. This is not research; it is a backward engineering study to support a conclusion that was agreed to by Forrester and SAP.
Our observation is that it is impossible to take money from vendors and produce a good outcome. Vendors sometimes reach out to Brightwork Research & Analysis to do some reports. I have yet to meet a vendor who cared about research. What they want is a report that shows them as “The Best!” That is why Brightwork Research & Analysis does not take income from vendors to produce content. It always ends up with the same thing: an inaccurate outcome and marketing masquerading as research.
The Problem: Thinking that Forrester is Focused on Accurate Research Conclusions
Every one of the Forrester studies that we review is rigged. Forrester is in the business of taking money from vendors and publishing whatever that vendor wants. Forrester both covers topics they don’t understand and have massive financial conflicts of interest.