How to Understand the IT Project Failure Ratio

Executive Summary

• Projects normally do not have a risk percentage applied to them in the project planning stages.
• Companies tend to prefer to assume that all IT projects will be successful and ignore IT project failure.

Introduction

It should go without saying that implementation success and implementation timelines are probabilistic. Enterprise software is generally complex and dependent upon several factors and the software itself. Furthermore, at least among executive decision-makers, it is well known that software implementation is a risky business. So why do so many companies behave as if project success is a preordained certainty? How do we know that this is a frequently held assumption?

Well, project and software selection as traditionally practiced relies upon the assumption that the implementation will be successful and that the probability of success is equal among all applications. However, a project or software selection without an approximation of the risk of a project makes is a poor way to make decisions about either projects or software.

The Actual Numbers of Supply Chain IT Success Ratio

I was shocked when I found this story when researching my book. I knew that many supply chain implementations failed to meet expectations, but the numbers below really surprised me.

Supply chain software, such as order management, inventory management, demand planning, and logistics applications, has been around since the late ’90s. But according to Forrester Research, in a recent survey of 22 business executives’ at large manufacturing and distribution companies, only 41% have realized a noticeably positive return on investment from their supply chain management systems.

AMR Research reports that about 65% of large enterprises have implemented supply chain software. Still, probably less than a fifth of those find that the software makes a huge difference to their businesses.

“When you start poking around, a company buys the software and is using it in just a few places,”

Says John Bermudez, senior VP of research at AMR Research.

“It hasn’t penetrated the company.”

To see my comparison of how SAP supply planning software compares to best of breed, see this link below. For demand, planning sees this link. For production, planning sees this link. For service parts planning, see this link.

Interpretation

I have multiple examples of when analysts like Forrester and AMR have been wrong. In fact, AMR Research, which Gartner now owns, is normally quite inaccurate. Some of this inaccuracy is because analysts are too far away from the subjects they cover. However, a second reason is vendors’ financial influence, substantial vendors that fund these firms. Therefore, I was shocked to see these results, which are entirely contrary, published by these companies.

It certainly does not motivate companies to buy and implement more enterprise software.

CNBC On Digitial Transformation Failures

Digital transformation is more than a buzzword for modern businesses—it’s a necessity for remaining viable as the future of work becomes increasingly mobile, agile and global. In fact, two-thirds of businesses recognize their company must digitize by 2020, in order to stay competitive.

In fact, CNBC is wrong here; digital transformation is the incorrect hijacking of a term used to describe the migration from analog systems to digital systems, as we cover in the article The Problem with Using the Term Digital Transformation on IT Projects. Before digital transformation becoming popular as a term around 2016, we had a perfect term called systems implementation.

However, CNBC has some important observations about implementations, and all that is necessary is to replace the term “digital transformation” with “systems implementation.”

Despite its critical importance, a surprising number of transformation efforts are failing, even at some of the world’s most profitable, innovative organizations.

Last year alone, companies poured $1.3 trillion into transformation initiatives — 70% of which was wasted on failed programs at companies like GE, Ford and Procter & Gamble. Among those that didn’t fail outright, only 16% saw improvements in their performance and ability to sustain change over the long haul. Even for digital-first industries like high-tech, media and telecom, only 26% saw success.

Much has been written dissecting the reasons for digital transformation failure — most experts have settled on people/employees, organizational culture and leadership as weak links. But few acknowledge the real common thread: communication breakdown.

The truth is, people aren’t the problem; it’s the organization’s failure to communicate effectively with its people that sets them up for digital transformation trouble from the start.

This is a very industry-friendly interpretation of the reasons for the failure. Brightwork Research & Analysis has provided analysis of false information provided by vendors and consulting firms to software buyers. However, CNBC receives income from vendors and consulting firms, so it is not surprising that they don’t even bring these points up.

Feeling Confident…For No Reason

Most supply chain IT projects I am familiar with start off very optimistically, with the team and management believing that their project will be successful. However, the research demonstrates that only between 25 to 35% of supply chain projects are successful. This is roughly the survival rate of U-Boats in WW2. Similarly, crews of U-Boats were also strangely positive that they would survive.

“We knew the statistics, it was not hidden from us. We all thought that we would be the 2 or 3 of 10 that came back. And we went about our business with this mind. – German WW2 U-Boat Sailor”

There appears to be something in human nature that makes us more optimistic than we should be about our chances of “survival.” In wartime, this may be a functional response to what are limited options. However, on projects, I believe taking a realistic view can help improve project outcomes. For instance, if the high failure rate on projects is known and disseminated, then less risky action courses can be selected from a series of more risky options. Without this knowledge, the more risk options seem more viable.

Like the lens of a camera, perception is designed to reflect the image of reality accurately. The market for distorted camera lenses is quite small. Realistic perception leads to more functional decision-making. Different cultures vary as to how much they promote positive thinking over rational thinking, with America being at the outer edge of what could be considered the spectrum of sanity. My evidence is that I am not aware of another country that has a publishing industry built around positive thinking.

While positive thinking can help overcome difficult odds, I can say that it can also become a crutch to excuse sloppy research and insufficient planning.

Towards the end of his military successes, Napoleon became increasingly unrealistic about the strength of his forces, the capabilities of his opponents, and his ability to win battles (as can be gathered from his change in garb from his military uniform to that of an emperor in the picture to the right). All of his military losses occurred after he mostly lost touch with reality. If he had maintained the more realistic perspective of his earlier effectiveness, his troops would not have to have been so positive.

Accurate thinking can keep a person and organization out of situations where so much positive thinking is obligatory. Which road ahead would lead to more premeditation and risk mitigation before the journey was begun? It is the role of project decision-makers to think realistically so that the decisions result in practical situations for the project and company. Thus the thinking needs to get more realistic and less confident as one goes up the hierarchy.

The Studies

There have been some studies on large IT projects since the 1990s, and they mostly show the same thing, roughly 30% of projects meet their objectives. Here is a small sampling of some of these studies.

Robbins Gioia Survey 2001

51% viewed their ERP implementation as unsuccessful. 46% of the participants noted that while their organization had an ERP system in place, or was implementing a system, they did not feel their organization understood how to use the system to improve the way they conduct business
https://www.it-cortex.com/Stat_Failure_Rate.htm

KPMG Canada Survey 1997

Over 61 % of the projects that were analyzed were deemed to have failed by the respondents. More than three-quarters blew their schedules by 30% or more; more than half exceeded their budgets by a substantial margin.

Chaos Report

The Standish Group research shows a staggering 31.1% of projects will be canceled before they ever get completed. Further results indicate 52.7% of projects will cost over 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg. The lost opportunity costs are not measurable, but could easily be in the trillions of dollars in the United States alone. On the success side, the average is only 16.2% for software projects that are completed on-time and on-budget.

Kweku Ewusi-Mensah

Kweku Ewusi-Mensah: A 1994 survey of 82 Fortune 500 companies by Kweku Ewusi-Mensah and Z.H. Przasnyski found that 44% of respondents had experienced “total abandonment” of an IT project and another 16% had experienced “substantial abandonment.” Source: Kweku Ewusi-Mensah, Software Development Failures.

Center for Project Management

Center for Project Management in San Ramon, Calif., estimates that 77% of projects blow their budgets, with an average cost overrun of 169%

In looking into this topic, we found the following observations to be instructive. These observations from experienced IT professionals indicate that there may be a fundamental misunderstanding about how projects progress and that some standard practices on projects may not be functional.

Except for extremely short projects, it is simply not possible to “get requirements right” and then devote time to dealing with those “correct requirements”, no matter how good the validation tools. Requirements will change over the life of the project. – Unknown

It’s important to place speaking and listening at the center of the enterprise

Interpreting projects as exercises in building and managing networks of commitments that people make to each, and not the imposition of a project plan constructed by an all-knowing project manager

Interpreting projects as an exercise in individuals coordinating their actions with each other

Managing projects as peer-to-peer, networked efforts, where everyone manages the project on the model of self-managing teams.

The core problem will not be solved by investing in getting requirements right up front. The core problem is outmoded theories and tools for project management. The GANTT chart was invented almost a century ago for controlling the flow of machinery on the shop floor. Other key artifacts are half a century old. The critical path today may be the inconsequential path tomorrow. The resources leveled this month will be unavailable next month. – Art Gould

Problem. When a company begins developing a new product, someone collects requirements from pro-spective users. This culminates in a Market Requirements Document for Engineering. One problem is that users often aren’t buyers. – Frank H

This research is for IT projects. I was not able to find any research on supply chain planning or advanced planning projects in particular. However, I do not have any reason to conclude that supply chains or advanced planning projects have significantly different success ratios. However, there is not sufficient focus on this, and a fundamental misunderstanding of the actual likelihood of success in preventing changes to the implementation process that needs to occur to increase the success ratio.

Estimation of Failure Likelihood

Companies by large do not estimate their probability of success before deciding which project to fund. Instead, they take the naive assumption that all projects will succeed, even though on average, IT projects only are considered a success roughly 50% of the time, and can be quantitatively shown to have a positive return roughly 35% of the time (we will get into the support for these values in a few paragraphs – but there are much more detail and complexity than is generally granted to these statistics).

There is little doubt that a high percentage of IT implementations fail. In fact, in my observation of projects as a working IT consultant – both in my specific area and in related areas, the failure rate is higher than is generally published. But this is for a rarely discussed reason. It is because all success and failure estimations that are tabulated rely upon self-reporting. That is, the company itself is assumed to know if their implementation was a success or failure. However, implementing companies often do not know, and most do not spend the time to measure whether their implementation was a success. Therefore, only the most obvious measurements tend to be used. This is explained in the following quotations:

“In addition Bradford and Sandy (2002) reported that 57 percent of the companies they interviewed had not attempted to assess the performance of their ERP systems owing to a lack of empirically effective evaluation models.

According to Parr and Shanks (2000) ‘ERP project success simply means bringing the project in on time and on budget.’ So, most ERP projects start with a basic management drive to target faster implementation and a more cost-effective project… Summarizing, the project may seem successful if the time/budget constraints have been met, but the system may still be an overall failure or vice versa. So these conventional measures of project success are only partial and possibly misleading measures when taken in isolation (Shenhar and Levy, 1997).”” – Measures of Success in Project Implementing Enterprise Resource Planning

Such obvious measurements would be if a system never becomes operational, or if the system makes it impossible to send out invoices, or otherwise causes a loss to the business. However, a system can be operational and can seem to work, but it can provide an output of poor quality and constant manual adjustment.

Factors to Consider

One can implement a new system with a more complex set of methods contained within it. Still, because either the software was poorly designed or improperly implemented, the output can be of no better quality or even worse quality than the system it replaced. When this happens, there is a powerful tendency to declare the system successful in any case. The business users may complain – but IT may simply respond that they don’t know how to use the system – and that the system is working properly. Sometimes, they will bring in an expert in the field to review the system, but they really want the system blessed. I know, I am one of those called in, and I have frequently found that when users are complaining about system output, they are typically correct. In any way you want to look at it, enterprise software implementation is a risky business.

The Probability of Success of Different Supply Planning Methods

The various methods of supply planning for advanced planning and scheduling are very different from one another. These methods, which are heuristics, allocation, and cost optimization, also differ greatly in their likelihood of being implemented successfully. Typically, during the software selection phase, a method is selected based on how well it meets the business requirements. Something which is left out of this analysis is the probability of success of the methods.

If one method provides all the things your business wants, but the company lacks the funding or expertise or sustainable orientation to bring up a solution, it makes more sense to select a solution that can be implemented. I rarely find that companies correctly estimate their abilities to bring up complex solutions. For instance, it is becoming increasingly common for companies to outsource support to India in the area of support. 

However, the outsourced model was never designed for complex solutions like supply planning APS solutions which are some of the most complex systems that a company has.

Resolving issues requires a detailed understanding of the issues, domain expertise, configuration history, etc.. It is not simply performing a password reset. Therefore, if a company wants to use outsourced support, not provide sufficient internal personnel for the implementation, etc…then the company should move towards an easier APS method and a higher probability of success.

How the Different APS Methods Compare in Terms of Probability of Success

Heuristic

Heuristics have a very high success rate. The SAP SNP Network Heuristic is about as easy to use as MRP but has extra settings that require some analysis and troubleshooting. The SNP Heuristic is extremely fast and can be run as many times as per day because it provides the same result if it is run for the overall network as it does if it is run for a single location or a single product location combination. The SNP heuristic can also be run interactively, which allows it to provide an instant update on the new situation.

CTM and Cost Optimization

However, there is a significant drop off in the success of the more complex methods, which include allocation and cost optimization. Few companies have success with allocation or cost optimization. This is because this method is complex. There are many screens of settings on each of these methods. These methods also require detailed configuration and master data maintenance in the area of resources, as most companies that select these methods are interested in performing constraint-based planning.

Additionally, allocation requires the development and maintenance of a table that declares which customers should receive inventory over others. In cost optimization, costs must be developed and maintained for the transportation costs between locations, the storage costs at a location, the costs of violating (dipping into) safety stock, the cost of production, and the costs of missing a demand.

All of this entails work, and this work must be appropriately staffed. It also requires a clear understanding and clear declaration of these policies’ policies and communication to the individuals who are maintaining the configuration and master data.

Conclusion

There are several different approaches to managing project risk. Some of the common approaches, which are looking for assurance from outside the company, don’t work.

I have told the story of the easily avoidable errors in both articles at Brightwork Research and Analysis and to friends and family members for years. When you talk to people within the industry, there is often a creeping cynicism, a “well, that is the way it is” mentality that often comes into the discussion. However, to outsiders how have not simply become numb to the poor management and uninformed decision making, there is typically more shock. IT implementations very much have a degree of repetitiveness to their errors.

References

https://www.informationweek.com/news/software/enterpriseapps/showArticle.jhtml?articleID=6900160

https://www.cnbc.com/2019/10/30/heres-why-ge-fords-digital-transformation-programs-failed-last-year.html

https://www.sandhill.com/opinion/editorial.php?id=67