Executive Summary

  • Software vendors love presenting biased case studies to prospects.
  • This article covers hidden problems with software case studies.

Introduction

Case studies are created and presented to software vendor prospects to convince prospects to purchase software and implementation services. Often little discussed is how accurate and reliable these case studies are.

This article is broken into the following sections.

  • Section 1: Important Features of Vendor Case Studies
  • Section 2: How Case Studies Tend to Be Developed By the Marketing Department of Software Vendors
  • Section 3: What Are Views of Some Decision Makers on Case Studies?
  • Section 4: How Our Case Studies Differs from Vendor Published Case Studies

Go to top

Section 1: Important Features of Vendor Case Studies

Now is an appropriate time to highlight some of the basic features of case studies. Everything that follows concerns vendor case studies, an important distinction, as case studies can apply to any area, so this list is only for vendor case studies.

  • Case studies are vendor-biased accounts of previous project implementations and outcomes presented in a one-off format.
  • Case studies have two perspectives, the vendor and the subject or customer, and there is no outside analysis.
  • Vendor case studies are almost always written from one point of view.
  • Case studies are plentiful on vendor websites, and most vendors have them.
  • Case studies are not a research product, and they are interpreted as marketing output.

Let us review a few examples of vendor case studies.

See the following screenshot.

This is a standard vendor case study, and it is on the vendor’s website and is written by the vendor. 

You can read the AWS Case Study to garner your own impressions. Indeed, you will have seen this format many times.

Reviewing One of the Best Case Study Websites

Google Cloud provides the best case study websites that we have seen.

So let us review their case study approach.

Google Cloud’s Case Study Overview Page

Google Cloud presents its case studies in a very appealing format. This is consistent with other Google pages we have reviewed, which we think have an excellent UI and give an immaculate and positive impression to the website visitor.

Google Cloud has a considerable number of case studies at their site, and they are well written, and the case study area has an appealing design.

Now, let us look at a sample Google Cloud case study.

Google Cloud has enormous resources to produce what looked to be around 1,300 case studies on their website.

Google Cloud also has a powerful Google brand working for it, which makes the case studies seem more credible than a typical vendor’s website. Google has lost substantial credibility with users due to how they censor their search engine and YouTube, but the brand for corporate services is still probably one of the best out there.

Google Cloud’s case studies are strongly differentiated from other case studies on the websites of other vendors in its number of case studies and its design and presentation.

However, what is not differentiated is the individual case studies themselves, as they are essentially similar to the other case studies we reviewed at the websites of many other vendors. Google Cloud uses the same fundamental approach as the case studies at other vendors.

Back in 2008, Formpipe’s ambitious product expansion plans brought the company to a point where its development capacity had to be drastically increased. Thus, they started seeking out a long-term nearshore supplier.

We started working on Platina with product technical review and competitive analysis providing a list of recommendations for improving the information system. Gradually the team has taken over the full-cycle development of the Enterprise Content Management product and now delivers a new major version of the product and a new service pack every year. – Sigma Software

And this excerpt is from a case study at CERN by Google Cloud.

Straddling the border between France and Switzerland, thousands of researchers are using some of the biggest, most complex scientific instruments in the world to examine the smallest particles in our universe. The European Organization for Nuclear Research (CERN), based in Geneva, is one of the world’s largest and most respected research centers, funded by 23 member states.

CERN has already redesigned some of its architecture to run on Kubernetes, the open source system developed at Google. Container-based systems such as Kubernetes, which focus on portability of services, allow IT infrastructures to deploy more quickly and scale more easily than traditional architectures. This made it relatively simple to incorporate Google Cloud for these investigations.

The core of the cloud deployment being evaluated through this project is Google Kubernetes Engine (GKE). One of the main features of GKE is the ability to scale up its clusters to very large sizes very quickly. GKE can automate administrative work, reducing administrative overhead and autoscaling clusters up and down as needed. Meanwhile, the addition of Preemptible VMs to GKE can provide flexibility in the way that workloads are handled. Preemptible instances are short lived and easily repeatable, making them ideal for small, repetitive tasks. – Google Cloud

Topic Transition

And this leads to the topic of how vendors develop their case studies.

Section 2: How Case Studies Tend to Be Developed By the Marketing Department of Software Vendors

Some observations about the many case studies we read before developing this research product and before writing this article are followed.

Observation #1: The Overriding Similarity of Case Studies

We just discussed the similarity between Google Cloud’s case studies and the case studies of other vendors. However, we found that the vast majority of the case studies at each vendor we evaluated were essentially similar. Even case studies from an extremely well-funded company, like Google Cloud, appear undifferentiated in just the written content than some of the most minor vendors.

Observation #2: How Video Enhances Credibility

Videos embedded in the case studies were very effective, and they made the case study seem more real. For one reason, it showed a higher commitment on the part of the case study customers as getting them to appear on a video set is a higher level of commitment than just offering some quotes.

Observation #3: The Problems in Comparing Case Studies

We constantly ran into problems with comparing the details of the different case studies. And beyond just reading different types of case studies, we reviewed multiple case studies within the same category of solution at Google Cloud. Still, we could not find many essential points of comparison. The way the Google Cloud case studies were written placed all of the analytical efforts onto the prospect, who would have had to extract information and create their matrix from the case studies. This observation was generalized to other vendors.

We have concluded this is a universal feature of case studies. The case studies we reviewed were unstructured, with some information appearing in one area of one case study and that same type of information being in another part of a different case study, even for the same vendor. There is no attempt to have a consistent location of similar data or even obtain similar data between case studies.

  • This makes case studies difficult to evaluate as part of a coherent whole with other case studies from the vendor, restricting pattern recognition.
  • This is because each case study is only designed to be evaluated individually.

Section 3: What Are Views of Some Decision Makers on Case Studies?

Another area of case studies that could use much more research is how case studies are the following…

  • The interpretation of case studies by readers.
  • The impact case studies have on executive decision-makers.

If we wanted to, we could easily add significantly to the research in this area. And a primary reason is there is so little existing research published on the topic. While the research in this area is puny, there are many articles on how to develop good case studies.

Furthermore, it is rare to find articles on case studies that discuss honesty in writing case studies.

Topic Transition

And this leads to the topic of how our case studies differ from vendor-produced case studies.

Section 4: How Our Case Studies Differ from Vendor Published Case Studies

Distinction #1: Annonymous Case Studies

Companies will not provide a case study with their name if the case study is not favorable. Our case studies are anonymous to allow the subject to tell us negative information making the case study far more accurate.

Distinction #2: Independently Written Coverage

Vendors or consulting companies have not input to the case study.

Topic Transition

This topic broaches how we charge for this service, so we will address that now.

Go to top

Conclusion

  • Case studies often come across as inauthentic and present an overly one-sided or overly optimistic description of the project.
  • Every single case study presented is a success and transformed the company fundamentally.
  • One of the biggest dangers of consuming vendor-produced case studies is that they come across as purely marketing collateral. The software vendors and their partnered consulting firms present them as representative of the experiences of their customers experiences with the software. They are selected from only the most successful cases.
  • The prospect also has no idea what was covered up or sugar-coated-in the case study. This allows you to learn from case studies that are accurate rather than reading marketing collateral.

How Software Buyers Should Turn the Tables on Marketing Controlled Case Study Development

  • Software case studies should not be entirely controlled by the vendor.
  • Software buyers can increase their access to quality information on previous implementations of software by obtaining case studies written without the input from the marketing departments of software vendors.