The Appleby Accuracy Checker: A Study into John Appleby’s Accuracy on HANA

Executive Summary

• For years John Appleby provided information about HANA to the market.
• This is a study that measure’s John Appleby’s accuracy.

Introduction

HANA was introduced in 2011, although little actually happened with HANA in 2011. In 2012 and particularly into 2013 and forward, John Appleby of Bluefin Solutions published large amounts of inaccurate HANA information. Appleby’s inaccuracies were so extensive that this study was created to catalog many (but certainly not all) of Appleby’s statements about HANA and to measure the average accuracy level of Appleby’s comments about HANA.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: We have no financial ties to SAP or any other entity mentioned in this article.

• This is published by a research entity, not some lowbrow entity that is part of the SAP ecosystem. 
• Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. As you are reading this article, consider how rare this is. The vast majority of information on the Internet on SAP is provided by SAP, which is filled with false claims and sleazy consulting companies and SAP consultants who will tell any lie for personal benefit. Furthermore, SAP pays off all IT analysts -- who have the same concern for accuracy as SAP. Not one of these entities will disclose their pro-SAP financial bias to their readers. 

The Method

This study is a document analysis of some of Appleby’s most popular and influential blog posts. That is one half of the study, while the other half is the reality of HANA. Brightwork Research & Analysis has performed the most extensive research on HANA, and this website has over 250 articles on HANA and S/4HANA (the ERP system mated to HANA). That is with the caveat of research outside of benchmarking. The entities with the benchmarking evidence around HANA are the database and hardware vendors. However, because of licenses and partnerships, these benchmarks cannot be published. In the one case where a benchmark was published, it was heavily redacted and rigged in favor of HANA, as we covered in the article The Problems with the Strange Lenovo HANA Benchmark.

Brightwork Research & Analysis is also one of the only information providing entities that cover SAP that is not financially tied to SAP (or to any other vendor) (for example, Forrester, Gartner, Nucleus Research, and all of the US-based IT media outlets are financially connected to SAP). This lack of financial relationship allows us to publish our analysis of accuracy without concern for retaliation on the part of the SAP.

Understanding the Retaliatory Measures Against Independent Research

The standard retaliatory measure being for the vendor to threaten to remove or reduce funding unless the offending analyst is fired, as Oracle requested against DeWitt when he published a benchmark that caused Oracle displeasure as Oracle was not ranked at the very top of the reference. Oracle went further in the DeWitt case by threatening the University of Wisconsin with no longer hiring its graduates. The position of firms like SAP and Oracle is that those that publish research that is not consistent with their marketing materials should be removed from performing research. And in the case of Oracle, they were intent on bringing this pressure against a major university.

The Method Used

The method was to fact check statements made in Appleby’s articles and rate each of the comments for accuracy. This resulted in an average accuracy.

What Statements Are Measured

Appleby is very well connected to SAP and has received a steady stream of information from SAP since he developed a relationship with SAP’s upper reaches. In his article 10 HANA Predictions for 2015, Appleby made “predictions,” but these were just things that development was working on at the time. We did not measure these statements for accuracy.

In some cases, Appleby made statements such as HANA worked with S/4HANA — which is true but would be like crediting someone for saying the “sun rises in the East.” Therefore, the study measured statements where something more than elementary and universally agreed-upon observations were made.

The Evidence for the Percentage Rating

• What you see below in the Appleby Accuracy Checker Table. This table (which is searchable, see the search box at the top) is designed to list the measured statements.
• The evidence that the statement’s accuracy percentage is found in the article link, which in each case links to a Brightwork Research & Analysis article where the statement is analyzed. Using a CTRL-F for the analysis article for the item you are looking for should bring you directly to the section you seek to find the supporting analysis.
• Because Appleby would not have made claims that were not cleared by SAP, and because most of the claims were also made by SAP, this study can also be used as a rough approximation of SAP’s accuracy on HANA. Our Study into SAP’s Accuracy covers that topic separately.

Conclusion

On average, for the statements measured, Appleby had a 6.43% accuracy to his statements around HANA.

Appleby had and continues to have a strong financial incentive to promote HANA. Individuals repeatedly say that just because they have a financial incentive to say something does not mean that this financial bias impacts their statements and positions. We have been told that paid placements don’t impact media content. We have been told that consulting companies that staff for SAP has no incentive to guide their clients to SAP solutions. And when we test that individual (or company), we find the opposite is true.

SAP customers continue to use information sources with 100% SAP bias when making their SAP investment decisions, regardless of all the evidence of the outcomes of financial bias. This is why we consider SAP (and most other IT areas) to be entirely non-scientific.