Quotes from the Book Software Wasteland - Brightwork Research & Analysis

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

Dave McComb’s book Software Wasteland is one of the best books on IT and IT waste.

Introduction

The quotes from Dave McCombs’ books are quite shocking. These quotes are so amazing; we created several articles to make them available to ourselves in an easily findable way to use them in the future.

Quote #1: The Major Problems with the Data Models of Software Vendors

All enterprise applications have data models. Many of them are documented and up to date. Data models come with packaged software, and often these models are either intentionally or unintentionally hidden from the data consumer. Even hidden though, their presence is felt through the myriad screens and reports they create. These models are the antithesis of elegant. We routinely see data models with thousands of tables and tens of thousands of columns, to solve simple problems. Most large enterprises have hundreds of thousands of these data models.

Quote #2: The Truth About Repetitive and Redundant Code

“Code Understanding Systems Systems that parse source code and reverse engineer its meaning have gotten better and better. At a first level of approximation, they can whittle down huge amounts of code to manageable subsets. The ground truth is that most of the code in a legacy application is doing little more than moving data back and forth.

When you strip away most of the repetitive and redundant code, there is often little left. A complex inventory system might have a few algorithms buried in it to set reorder points, calculate efficient order sizes, and flag unusual demand patterns. This is augmented by some very simple logic around re-determining average item cost based on recent receipts and choice of costing methods.”

The ability to break down and categorize code is critical to refactoring code and taking control of the coding environment, leading to the more efficient management of the code.

Quote #3: Why Do Rogue IT Systems Get Started?

By the way ,the presence of a rogue system such as one developed in Microsoft Access or Excel, might be a good proxy for change metrics. No one wants to build a rogue system.

People build rogue systems because they have a need that would be too hard to implement in the existing system. About half of the rogue systems we have examined are systems where someone wants to view existing data using a category that doesn’t exist in the base data, and then do some simple calculations. Keep in mind that if the coset of change was low, there would be no need for rogue systems.

What vendors and consulting firms and IT departments do not want to admit is that the systems they offer the business normally do not meet all of their needs.

Quote #4: How to Identify Waste in Software Projects

“Figuring Out the Quagmire

With each application get two statistics:

Schema Size: Number of tables plus total number of columns for relational systems, number of classes plus total number of attributes for object-oriented systems, number of terminal elements plus total number of attributes for XML based systems, etc..
Number of Lines of Code: The number you have access to and are maintaining and don’t count code you can’t modify.

This will give you the factors you need to assess the total complexity of your information infrastructure.

Interfaces

Very few companies have a good inventory of the interfaces they have in place, which are costing money to support and that are hampering their attempts to migrate systems. There are several approaches to this step, depending on the number and complexity of your interfaces. At more than a few hundred apps, you will need some automated support. Several software firms can help with this, including Eccenca, GlobalIDs and Io-Tahoe. After you have profiled your data and determined that size different systems have identical histograms for certain data sets such as customer or product lists, the next step is to instrument the systems in real time and figure out which one is getting populated first and by implication, which ones have interfaces that are being fed.”

It is beneficial to have a proven method of identifying software waste, and the method provided in Software Wasteland is an excellent place to start.

Quote #5: Who Would Lose From Less Software Waste?

“Most of the established application vendors now offer cloud hosting but they haven’t changed their licensing or integration economics, so it only affects a small part of the equation.

The application software and the system integration industries may be the most threatened.

In the end game, they are slated to go from about $400 billion per year for each of the two industries, to asymptotically close to $0. Small businesses and the digital divide companies (those founded after 2000) will fare the best, as they have the least invested in the quagmire.”

All of the current entities that benefit from the current system of waste would stand to lose.

Quote #6: How To Maximize IT Consulting Profits By Not Implementing Systems

(One of the main problems with our industry is that there is far more money to be made by being incompetent than there is for being competent. There are still far too many contractors who make far more money not implementing systems that there are contractors that can implement productively.

This is a constant issue in IT consulting. The largest consulting firms have the system to don’t actually need to bring systems live to be paid.

Quote #7: Why Is Software Waste Worse in the Government

“The following factors conspire to make the situation worse for the government.

1. Difficulty in hiring top talent
2. Overly restrictive purchasing procedures
3. Contractors that specialize in gaming the system
4. Willingness to accept mediocracy
5. Budget cycles
6. Lack of incentive

An entire sub-industry exists due to the size of the goverment software industry. Pejoratively called “”beltway bandits”” from their headquarters near the I-495 beltway around Washington DC, these firms specialize in servicing the government markets.

There is nothing wrong with this per se, but what has happened is these firms have specialized in winning, expanding, and extending government contracts far more than they have specialized in performing them. This is because there is very little incentive to perform, and a great deal of incentive to capture (as they call winning a bid).

The beltway bandits know how to answer questions in a way that the contractors are more or less forced to be awarded maximum profits. They specialize in protesting when they don’t win. Many government employees are fearful of protests, not only because it delays their projects even further, but because it holds them to scrutiny.”

“Another popular practice is getting involved with an agency in advance of an RFP and making sure there fare requirements that only they can satisfy. This generally favors firms with large marketing firms who can dedicate resources to the long-term cultivation of contracts inside of the agency.

This has lead to extreme conservatism. If you don’t believe you can hire and retain talent, you are unlikely to embark on custom development projects. If you believe that your contractors will game the procurement, you will define projects that have little risk and require little skill.”

It is a constant feature in government entities that they get defrauded by large IT vendors and consulting firms. In fact, they have special divisions with their organizations that know exactly how to vacuum up taxpayer dollars.

Quote #8: How Good is the Logic of Buying Packages Because One is Not in the Software Development Business?

“We’re not in the information systems business.”

Many companies use this line to explain why they should not build (or should not even run) their own systems. This line was used to justify a great deal of outsourcing of information systems. Even if you do not create software, you use it. The act of using software, especially software that mediates your key processes, puts you in the systems business. If you contemplate changing your business process, this necessitates changing your information system. If you don’t have the ability to change your own software, you are at the mercy of whoever does.

We understand why people say they are not in the information systems business. It’s mostly because they don’t want to be in the information system business. Additionally, it’s often because they were not very good at information system building and information system implementing. However, wishing to leave the systems buisness is not the same as leaving the systems business.

IDC has come to the same conclusion.

“”Enterprises are turning away from traditional vendors and toward cloud providers. They’re increasingly leveraging open source. In short they are becoming software companies.”””

The argument that “we are not in the software development business,” as pointed out by Dave McComb, implies that one can stay out of the software development business. The fact is packaged solutions, in most cases, will not cover all of the requirements of a company. This means the packaged applications have to be customized, and at some point, it is more cost-effective and sustainable to develop applications internally. This is not to say companies should not leverage packaged solutions where appropriate. But the advice on the part of major consulting companies and vendors is to the degree possible to eliminated internally developed applications.

Quote, #10:How Effective is the IT Logic of One Throat to Choke?

In the “one neck to choke” model, we would have said: “You’re responsible for everything, hire whichever subs you want, but you have to make it look good.” I’m sure, has this been an option, the price would have ballooned.

First, the general is only going to hire the best contractors (can easily get cost inflation here), is going to pad the estimate to make sure they can handle any claims, and will be rigorous about the scope. My guess is this sort of arrangement could have easily doubled the cost of our basement projet. The cost gets away from you because there is no inbuilt market for each part of the project. The way we did it, we could cost different concrete treatment treatment subcontractors, and if they looked like it was getting unreasonable, we could shift to wood or carpet.

Once you go for all-in price, you lose a lot of opportunity for constructive change. My observation is getting a sole source for a large project is almost guaranteeing it will cost five to ten times what it would cost if it were more proactively managed.

Our experience is that “one throat to choke” never works. First, big firms like Deloitte or IBM, or Accenture don’t get choked. Instead, they choke their clients. They are experts in robbing and choking their clients. These giant firms are both not very good at implementing and are too large to control. These firms are also highly trained and competent in defending themselves in court.

Quote #11: The Problem with Measuring Developers by Lines of Code

This was also the case with software. The primary early metric was LOC (lines of code). The belief was that a program twice as long must bave solved twice as difficult a problem. Moreover, a project manager could estimate how large their system was (in LOC) and measure progress to that goal.

Only this backfired massively.

Once programmers knew they were being measured by LOC, they started writing more and more LOC. Reusing a subroutine was detrimental to getting credit for writing more lines. I recall watching project managers recoil in horror as their LOC counts exploded and the project hardly moved any closer to completion. The number of lines of code is barely correlated with the size of the problem being solved, it is highly correlated with something else: defects.

This is really something that should have been caught earlier. As soon as the managers showed no relationship between code lines and project completion, this method should have been dropped for measurement.

Quote #12: Will Data Lakes Solve Data Warehouse Problems?

“The “”data lake”” approach took big data techniques and applied them to the frustrating parts of the data warehouse solution. Rather than conform the source data to the reporting structure, the data lake approach says, just lay the data down as it is. Then use big data techniques to find and correlate it at read time rather than write time.

In the early days of data warehouses, and especially in the being economical about reusing “”conformed”” dimensions. These days, data warehouses have thousands of tables, many of which resemble the tables of the applications they were extracted from. “

Actually, the new information coming from data lake projects is that they have gotten out of control. The major problem is that the data lake is often turned into a data dumping ground where the idea is that value can be obtained from the data lake later, as long as the data is simply accumulated.

Quote #13:The Pipe Dream and Misunderstanding of Big Data

Big data is primarily techniques borrowed from Google, Facebook and many other early adopters of parallel processing against data sets that were too large to be processed sequentially.

Many people think big data is the systems we have with more data. They think that a data warehouse is a petabyte of data is “big data.” It is a lot of data, but big data is more of a style of interacting with the data than an amount of data.(emphasis added)

In big data environments, the data is spread over many machines. The algorithm to be applied is written in a way that it can be massively replicaed. Because it is massively parallel, it can solve problems that couldn’t be considered in traditional processing patterns.

The fact is that few companies are really capable of even pulling off Big Data projects. However, the consulting companies have presented a fiction that Big Data is available to any company — with their help, of course.

Quote #14: Will Cloud Services Make Technical Debt Worse?

“The rise of cloud computing was about being able to spin up a server in minutes (as opposed to months) and to pay only for what you used. If you were able to pay only for what you used, and contractually you could cancel at any time, you were able to put the costs in the department’s expense budget.

However, think about this: what is letting everyone in your organization with a credit card in card and expense budget to become a systems implementer doing to do to your overall system architecture? This is making the costs of integration higher. The mere fact that data is in a SaaS vendor’s database makes it far less likely that it is integrated with your data. The net/net of this is a brewing storm of “”integration debt.”” “

This is rarely discussed in conversations around cloud service providers like AWS and Google Cloud. And then, of course, there is the combined issue of cloud service provider usages and SaaS usage. However, there are ways of controlling this.

When we developed Brightwork Explorer, we created a simple SaaS application that allows for easy import and export. But most SaaS vendors seem to have the customer input their data, but vendors like Salesforce made it very difficult to remove that data. This improves account control and is called “one way SaaS” or “one-way cloud.”

Quote #15: Has Agile Made the Problem of Technical Debt Worse?

Our observation is that agile has not addressed the core problem at large enterprises. While it makes individual applications smaller, more economical and easier to change (all good things), it has not addressed the bigger picture of multiple inconsistent applications and their databases. If anything it has made the problem worse. By encouraging small teams to solve individual problems, it fosters applications proliferation.

Agile has bombed, and secondly, what most companies call Agile is simply a method of getting more work out of developers. There is little about Agile that focuses on how any one application connects to another application. The majority of the focus of Agile is simply how to more quickly and with feedback develop applications.

Quote #16: Failures and Problems with Offshore IT

“Offshoring hasn’t worked out of well as firm intended. Most firms intended. Most firms have discovered that the offshore workers need much more literal specs to be effective. There is something to be said for literal specifications; in some cases, this improves communication. For many types of systems, though, the cost to make the requirements literal is greater that the cost to implement. We’re spending more money to get a watertight spec than can be implemented without error; than the original cost would have been. This is masked because the cost of implementation has in fact gone down, if you only count the offshore component. However, in many cases, the total cost has gone up.

Additionally, many of the popular offshoring locations have been experiencing very high turnover. As members of India’s technology class graduate into their first jobs with offshore companies, they rapidly develop marketable skills. This often leads to job hopping. Turnover in Indian programming shops ranges from 25 to 40% per annum. We have several clients whose projects have been massively affected by the high turnover of their offshore IT groups.

What we have seen is the focus on unit costs (labor costs for development primarily) tends to encourage more development (more modules more lines of code) when a more economical architecture would have far less than both. “

IT outsourcing allowed managers to take credit for direct cost savings, but the overall quality of service delivered normally declined. Companies ended up dealing with India in most cases, which is a highly corrupt country. Lying is the normal way of functioning in India, and therefore information that comes from India has low accuracy. Outsourcing has enabled the largest consulting companies to engage in labor exploitation and has lead to the rise of truly terrible companies like WiPro, Infosys, and Tata Consulting Group.

Quote #17: Why Companies Hire Awful IT Consulting Firms

The cost of professional services has come to dominate information system costs. This has lead to two reactions: outsourcing and offshoring. The two co-occur a lot and people have taken them to be synonymous. When you hire people and pay them a salary, it is your problem to make them productive. Making IT people productive is notoriously difficult. Many business people have opted to pay for results, and make someone else responsible for getting the productivity. This is the essence of outsourcing: replace employment contracts with performance based contracts.

“Moreover, while theoretically you could cancel one outsourcing agreement and replace it with another, from a practical standpoint there were many problems with this. The outsourcers were typically better negotiators, and usually negotiated very generous cancellation clauses, which meant that in effect it was harder to get out of an outsourcing contract than it was to fire a bunch of your employees.

Many outsourcing contracts have provisions for add-ons, but at prices pegged decades ago. One of our clients has an outsourcing contract that requires that all additional data storage be procured through and from the outsourcer, and that the equivalent of a small server will cost $20,000 / month. These were credible charges in the late 1990’s but a small server can be deployed these days for about $100 per month, or in the cloud for $20 per month. There is now a strong trend to “”in source”” or bring back into the fold the systems that had been previously outsourced, a clear indications that this trend did not solve the fundamental problem, and likely cost many firms years of potential progress.”

The value offered by consulting firms is normally atrocious. Still, the hiring companies seem to think that they cannot themselves manage IT resources for many projects and want to be assigned to blame external entities.

Quote #18: What Are the Origins of the Data Warehouse?

“Up until the late 1980’s if you wanted data or reports, you went to the application and used their reports or report writers. At some point, astute companies noticed that most information needs had to be sourced from many systems. Hence the data warehouse was born. And industry was born. Complete with arguments: should we follow the Kimball method or the Inman method? Should we embrace “”star schemas””? Should we load data marts into the warehouse or populate the data marts from the data warehouse? Should we have an “”operational data store.””

What the data warehouse did, that ERP and SOA did not, was combine data from many different applications and allow people to report against it. Ultimately data warehouses sunk under their own weight.

As more and more data was being extracted from the source systems to feed into the data warehouse, more and more tables were being created in the data warehouse. We have clients that have over 20,000 tables in their data warehouse. At some point this hardly seems like an improvement over just accessing the source files, other than in the process, the database access has been homogenized. In a large organization, bringing a new data set to the data warehouse can often take months. The cost of change has gotten out of control and has lead to the popularity of the “”data lake””, which we will discuss shortly.”

Like ERP systems, the data requirements around data warehouses also grew out of control. Data warehouses are known for having very low productivity. And now, multiple approaches have been recommended to address these issues. All of this is before even considering the effort companies have put into their Big Data projects.

Quote #19: How Complexity of Data Management and Models Continues

Since extending the existing table may mean rewriting code, we often create a new table to handle the extra columns.

The net effect of making many small changes such as this over many years is profound.

Most relational databases that we have examined are orders of magnitude more complex than they need to be.(emphasis added)

That is, the total number of tables and columns are 10 to 100 times greater than necessary if the system was implemented efficiently in a modern environment.

“By the way, there is also a perverse incentive for keeping the complexity. If it takes years to master the data model for an application, anyone who has mastered it has job security. If it was developed in house, the company becomes dependent on these key resources. If this is a package, the knowledgeable consultant has an attractive career.

There is a belief that the application code that accesses the relational database contains a great deal of business logic or algorithms. It does not. If you look long and hard at application code that accesses relational databases, you will find that 90% or more of the code is very repetitive and simplistic. There is some logic that says things like “”if the value in the “”into-go-down-marker”” is “”1″” then subtract the balance from the on-hand value”” This may pass for business logic, but it is working around bad design. “

Relational databases are often greatly extended in complexity. This usually comes from not having a solid data-centric model before simply adding on all of the applications’ data models from the many vendors that a company buys from.

Quote #20: How Consulting Companies Constantly Rob Their IT Customers

“Are the people who charge $1 billion for a $1 million system crooks, idiots or heros?

If you beleive that these simple systems are complex and should cost $1 billion, then a firm tha can complete it for $1 billion is a hero. I find no evidence that these projects are even within two orders of magnitude of their “”shoud cost”” numbers.

In other words are these are the DeBeers of the software implementation business (the DeBeers know that diamonds are fairly common, but have managed for over 100 years to control the supply and pump up hte demand, such that the jewelry grade diamonds command at least 10 times their free market value.)”

These companies like Accenture, IBM, Wipro, Deloitte, and many others are extremely experienced in ripping off their clients.

Quote #21: Large IT Project Over Budget Benchmarks from Software Wasteland

“Large system implementations

Initial budget $100 million
Initial time frame 3-5 years
Cancellation rate 50%
The “”successful”” projects are typically 50% to 100% over budget and years over schedule

That is the norm, and then yes, a few will run over by 1000%, just as a few bad apples will spoil the bunch.”

This shows the constant problem with projects is that they nearly always go over budget and schedule.

Quote #22: The Project Management Problems with HealthCare.gov

“The HealthCare.Gov site was written with 500,000,000 lines of code. Evaluation of what the site did concluded that even if the code were verbose, it should have been no more than 10,000 lines of code.

The HealthCare.Gov site was written with 500,000,000 lines of code. Evaluation of what the site did concluded that even if the code were verbose, it should have been no more than 10,000 lines of code. With Health Sherpa, a team of 3 or 4 in two to three moinths had build an equivalent site to HealthCare.gov which hundreds of professionals had spent years laboring over, at a cost of $2 billion.”

This is a very typical project when a major consulting firm implements it.

Quote #23: How to Fight for the Data Centric Approach?

“McComb: No firm that has achieved this wants to fall back, but some do. What often happens in a management change or an acquisition will result in management that is unaware of the economics of information systems. Worse, they often have deeply-held erroneous beliefs, such as the belief that implementing package solutions is always the most effective thing to do (emphasis added) (it is occasionally, but few managers predict how often it is more expensive than the build option, and none consider the added complexity to the firm as a whole and the impact on systems integration).

The main defense is not technical, but political. You need to over communicate the value that new approaches bring, and make sure people are aware of the complex trap they have avoided and are in danger of returning to. The Data-centric approach is still far from mainstream, and will have many detractors, inside and outside the firm. Many people will be threatened by this. They will not go quietly into the night.”

Very few people outside of data management even understand the data-centric approach. The application-centric method has many powerful advocates, the consulting companies, the software vendors, and even IT departments greatly influenced by consulting companies and software vendors.

Quote #24:How Much Complexity is in Legacy Systems?

McComb: The ground truth is that most of the legacy systems that exist have very little essential complexity and a huge amount of accidental complexity. To put it another way, a legacy system with 10 million lines of code probably has a few thousand lines of what anyone would consider to be algorithmic code. There are probably tens of thousands, maybe even hundreds of thousands, of lines of validation and constraint logic that could easily be replaced with parameters and model driven development. The problem is this trivial bit of complexity is marbled throughout the ten million lines of overburden.

Overburden is the low grade ore you have to remove before you can start open pit mining, and seems to be an apt metaphor. If you use reverse engineering software to find and isolate this, you’ve done yourself a great favor. Unfortunately, often reverse engineering is used to help migrate from a legacy system to a neo legacy system.

This is why merely estimating the number of lines of code overstates complexity.

Quote #25: The Use of Graph Databases According to Software Wasteland

The digital native firms, such as Google, LinkedIn and Facebook, have all their core data implemented in graph databases (such as Google’s Knowledge Graph and Facebook Open Graph). Graph databases are inherently more flexible. You do not need to have all your schema defined ahead of time (as is the case with relational) and every class does not need to have the same set of properties. Semantic models can be implemented directly on standard compliant graph databases (Triple Stores) such as Allegrograph, MarkLogic, Virtuoso, Stardog and Oracle 12g.

Some of the elite firms are following a data-centric model — and according to Software Wasteland, they are often using graph databases that provide high degrees of flexibility.

Quote #26:Why Application Centric Approaches Continue According to Software Wasteland

McComb: I attribute the continued existence of application-centric thinking to four causes: habit, unawareness, perverse incentives and lack of discipline. Habit might be the biggest reason, people have been getting systems approved and built for decades the application centric way, and old habits die hard. Many people are unaware that there is a different way, and you can’t change if you don’t know. The systems integration and application software package industry unfortunately has very perverse incentives.(emphasis added) They make far more money implementing yet another system, no matter how inefficiently.

Finally, many people don’t realize this is not something you buy, or a short term project you implement. This requires a great deal of discipline over a long period of time (not a lot of money, but a lot of continuity). In the case studies I interviewed for my next book, all of them took at least five years to get to a reasonably mature data-centric architecture.

Application-centric models continue because this is how software vendors go to market and how they influence their customers. Each vendor wants the customers to accept the model of each of their applications. Vendors tend to care very little about the technical debt they impose on their customers.

Quote #27: Lying About Being Data Centric According to Software Wasteland

“McComb: Many companies, when asked, would like to be data-centric. Some even profess to be data-centric. Very few are. What happens is they may build an individual application that is data-centric (that is the data was designed first, and process and functionality applied later). But if you build (or buy) 1000 data centric applications your firm will have 1000 centers, which is to say, no center.

The problem is, with an application centric approach in your mind, every problem looks like another soon-to-be siloed implementation project.”

Dave McComb is correct. The amount of software waste is enormous, and it is promoted by the largest firms, firms like Accenture, IBM, Gartner, SAP and Oracle, and many others. This waste’s major centerpiece is the proposal around using packaged software at the exclusion of internal custom development and using the largest (and most corrupt) software vendors and the largest (and most corrupt) consulting companies. This makes the companies application or vendor data-centric rather than customer data-centric.

Quote #28: The Enormous Waste in IT According to Software Wasteland

Dave McComb: It bothers me that we are rewarding bad behavior in the Enterprise Information Systems industry. The worse a system integrator or application software company’s ability, the more money they make, provided they can convince their clients that anti-productivity is necessary. If you can convince a client that a project should cost $100 million, you have a nice several hundred person project for several years. If it were widely known that such a project could be done by five people in six months, this waste would evaporate.

“McComb: There are an embarrassing number of cases where systems cost more than 1,000 times what they should. Most people have heard about Healthcare.gov. Some know that to date it has cost $2.1 Billion (against an original budget of $93.7 million). Even fewer realize that it could have been built (much better) for under $ 1 Million, which is exactly what HealthSherpa did. Healthcare.gov ended up adopting many of the design elements that were developed at HealthSherpa.

The Canadian Firearms Registry was sold as a $2 Million project ($119 million in cost and $117 Million in offsetting revenue). It cost $2 Billion, and registered 8 million guns before being decommissioned.

Most of the best stories are in the Government, partly because they are easier targets to the firms that prey on them, but also because so much of it is in the public record.

Private firms also often pay far more and often get very little from their large application and integration projects. We know of two large firms that launched a $1 Billion and a $250 million integration project that ended up with nothing to show.

There once was a big row about NASA paying $600 for a hammer (it later turned out that they paid $15 for the hammer and allocated some of their internal R&D expenses on a per item basis, rather than a per dollar basis which made it appear worse than it was). The worst excesses of traditional contracting don’t come close to what is becoming routine for systems contracting.

Most individual application projects are on the order of 10 -100 more expensive than they could be. But the problem is more systemic than just fixing an individual application project; the ecosystem that the projects live in is the problem.”