A Study into the Accuracy of SAP

Executive Summary

• This is the only independent research into SAP’s accuracy.
• This article documents that accuracy of SAP versus Brightwork Research & Analysis.

Introduction to the SAP Accuracy Study

In most research, the topic is introduced, and the background is provided, and the data is provided second. However, this research serves two purposes. One purpose is to serve as a long-term resource for people and companies interested in fact-checking SAP. For this reason, we have placed the data first and the explanation second.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: We have no financial ties to SAP or any other entity mentioned in this article.

• This is published by a research entity, not some lowbrow entity that is part of the SAP ecosystem. 
• Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. As you are reading this article, consider how rare this is. The vast majority of information on the Internet on SAP is provided by SAP, which is filled with false claims and sleazy consulting companies and SAP consultants who will tell any lie for personal benefit. Furthermore, SAP pays off all IT analysts -- who have the same concern for accuracy as SAP. Not one of these entities will disclose their pro-SAP financial bias to their readers. 

• If you are here just for the fact-checking, the table you wish to query is below.
• You do not need to type in the entire term. Type in the first few letters of the word you seek, and the list will begin to filter immediately.
• If this is your first time seeing this research, please click the Jump to Research Explanation link below to be taken to the full explanation of the study.

Jump to Research Explanation

A second purpose is for those with a more academic interest in SAP’s accuracy.

*Scan the list, or use the search box in the upper right above the table.

*Any cell with N/A means that either SAP or Brightwork did not make a statement about the line item.

Overall Accuracy Calculation and Future Accuracy Forecasting

The following is a straight average of all of the items listed in the table. This is a record of statements and predictions made over decades. In statistical forecasting, the historical forecast error is typically used to estimate future forecast error.

Following this precedent, it should be considered a useful accuracy percentage to apply to future predictions by SAP.

What should be clear is that SAP has a long-term issue with accuracy. This is the only published study in SAP’s accuracy. It is not generally understood how low SAP’s accuracy is by those that work in SAP or even those vendors that compete with SAP. 

A Recognized Advantage Versus SAP

Brightwork had the advantage of making observations years, in some cases decades after SAP initially made its statements/predictions. However, Brightwork performed well even for recent SAP projections. Brightwork Research & Analysis has made several predictions that contradict both SAP and the overall market. At this point, not enough time has passed to determine if these predictions can be considered to have come true. Therefore they are marked as TBD.

To be accurate, any entity must value accuracy over other objectives. It should be remembered that companies are generally said to maximize profits, not to maximize the accuracy of their statements to customers. SAP’s score illustrates the extremes this can take. 

Is Accuracy SAP’s Concern?

SAP is trying to drive revenues and is very obviously not attempting to be accurate. Individuals within SAP may or may not believe what they are saying. The study is not engaging in analyzing what statements SAP representatives know as false when they say them. It only analyzes if the statement turned out to be true.

Research Background

For obvious reasons, neither SAP nor the massive army of companies and people that make money from SAP and cover up for them is not fans of the research we do at Brightwork Research & Analysis.

We have been accused by SAP and SAP IT departments of the following:

1. Having an ax to grind (which we responded to in this article).
2. Of biting the hand that fed us (which is the opposite of the previous argument, which is having an ax to grind). This argument (biting the hand that fed us) we agree with does not make our research untrue, merely disloyal. We guarantee no loyalty to any software vendor or any consulting company or entity, whether we have financially benefited from them or not.

SAP’s inaccuracies are repeated through a network of entities that do not care about accuracy much more than SAP. This topic is the subject of future research, with IT media entities also being scored for their accuracy on SAP. 

Partners in Inaccuracy?

What SAP and other SAP proponents are careful not to do is address the conclusions of our research and analysis. SAP prefers to direct people to sources of information like

• ComputerWeekly
• Forbes
• CIO Magazine
• Gartner and other media entities and IT analysts that SAP pays.
• Or to Deloitte and Accenture, and other SAP consulting partners, entities with huge businesses built around billing for SAP consulting.

Therefore, according to who SAP recommends that their customers listen to, the universal feature is that the sources in some way have a financial tie to SAP, which we covered in Which Sources Does SAP Consider Credible?

SAP seems to have a precise definition of what is credible. The source seems to line up very carefully with sources that unthinkingly repeat what they say and where the money is involved for the entity that obeys SAP.

So how is a person trying to get a straight story to determine who is credible?

Determining Source Quality

In our view, a compelling way to determine source quality is to check their track record. Therefore, in this article, we will compare SAP’s track record to our own to see which is more reliable.

Background on Accuracy Measurement

We provide a relative accuracy measurement as part of our Honest Vendor Ratings.

We think that this type of measurement of accuracy is essential to decision making in IT. Our book Rethinking Enterprise Software Risk is highly based on evaluating the accuracy of sources.

However, this is the first time we measured, calculated, and documented a vendor’s accuracy in anything like this level of detail. It is a lot of work, and we don’t know anyone who does it. First of all, to do something like this, you would need to have financial independence from the source, which rules out virtually all media entities from doing anything like this. Secondly, there is no way of monetizing something like this. Media and IT analysts primarily make their money by promoting software vendors. One can’t collect money from vendors by challenging their assertions. It also tends to make a lot of people angry at you to boot.

Tracking Predictions and Outcomes

SAP’s and our predictions can be easily tracked through the published articles where they make the prediction. The determination of whether that prediction is accurate is a bit more tricky. But in most cases, we have informative articles that explain what the eventual outcome was.

Another trick in measurement is that some predictions, for instance, that Brightwork or SAP made, have not come true yet. But it does not mean they will not come true. In those instances, we don’t give a complete rating but instead, explain the prediction’s current state.

Finally, some predictions cannot be declared as true or false; they are not binary. Some predictions worked out to some degree. In many cases, it makes the most sense to provide a percentage.

Here is how we scored the outcomes of predictions.

1. A zero percent rating would mean that the prediction did not come true, or the exact opposite happened.
2. A 75% rating is that the prediction mostly came true.
3. A 100% rating means that the prediction entirely came true.

This is not a “perfect science,” but (unless you work for SAP or an SAP surrogate), we think the readers will find that this is entirely a beneficial exercise. One might question the entity rating itself against SAP and the primary entities that provide information about SAP. However, the results were so noticeable that we have every incentive to grade ourselves harshly.

Something we also have planned in this area is to measure the accuracy of the statements of IT media entities concerning SAP.

Conclusion

There you have it. So given this comparison, Brightwork Research & Analysis beats SAP by a substantial margin in predicting SAP.

The first observation is that SAP has a deficient accuracy level for its statements. Almost none of the projections that SAP makes come true.

How are we able to have such a high accuracy level versus SAP?

Well, first of all, SAP is a very aggressive company. Secondly, most of the SAP executives that make these claims are not attempting to be accurate; they are trying to hit sales goals.

We often receive negative feedback from SAP proponents on the research that we do. SAP proponents don’t seem to have an issue with SAP’s statement or prediction accuracy, but they seem to have a problem with an independent entity that checks what SAP says.

In forecasting, there is a technical definition of bias. Bias is when the forecast is consistently positive or negative. Bias is not the same thing as forecast error. You can be wrong 100% of the time but still not prefer if your mistakes are both positive and negative. (We cover the topic of forecast bias in the article How to Best Understand Forecast Bias.

SAP’s Constant Flow of Inaccuracy

From the observations in this study, SAP has very close to the highest possible error and the highest possible bias. That is, they nearly always overstate or have a positive spin on their projections.

Interestingly, we have been accused of being biased by SAP proponents. However, upon performing this research, not only do we not display any bias, but we also barely show any error. You can’t have a preference if you have almost an error.

Forecast bias is an appealing criticism because it is a charge that can be leveled without bringing evidence. When we level the charge of bias, we provide the reason. The most common reason we provide is financial bias or, in some cases, both financial bias and contractual bias. Such as in the case with SAP partners that are both financially incentivized to support SAP and contractually limited in what they can write about SAP as per SAP’s partnership contract, as we cover in the article The Control on Display in the SAP Partnership Contract.

Judgment or Prejudgement (Bias)?

Secondly, in general argumentation such as on LinkedIn forms, the term bias is often misused to mean a view that is not in agreement with the other debater. Also, the term bias is applied inaccurately as the term prejudiced. Being prejudiced is to prejudge something. It is not to judge something. The term prejudiced is inadequately applied when used as an ad hominem to describe when a person dislikes or disagrees with something they have sufficiently tested or experienced to render a judgment. For example, if one writes a movie review without seeing the movie, they prejudged the movie. But if they saw the entire film and then wrote a critical review, it cannot be said that they prejudged the movie. That is not an accurate criticism. They judged the movie.

Overall, SAP proponents have a different definition of bias than the actual meaning. The definition of bias, according to most SAP proponents, is whether you agree with SAP. If you agree with SAP, then you have no bias. If you disagree with SAP, then you have a bias.

Jump Back to the Accuracy Table