We recently helped a client with a project choosing a supplier. This raised some interesting management and strategy questions, so we thought a heavily anonymized post might be interesting here.

The client needed to choose a supplier for a fairly important system. This was more than just choosing a single component, it was a whole system. The purchase decision would have important implications for the performance of their whole product, and would change engineering requirements depending on the final decision. This this was a whole system it came with all sorts of important contingencies, dependencies and support needs. So the customer needed to perform a fairly in-depth analysis, not a just quick table showing price and performance, selecting the lowest bid.

One of the key criteria was the supplier’s ability to support this product many years into the future. Their end product had a life cycle of over a decade, so the client wanted to understand if each supplier would still be in business a few years from now. None of the suppliers were start-ups, they were all established companies with a long history. They were all profitable with comfortable-looking balance sheets. However, the system we wanted was a new product category to all three, so the question was not so much would the company survive, but would the product line survive? If the question had been company survival we could have used credit scoring, a relatively simple process. Instead, we were faced with a corporate management problem, which is probably why we got engaged for the project.

We started with the assumption that all these companies have the resources to survive, but needed to understand their ability to invest in a product that is probably not going to be profitable for a couple years. Which company had the fortitude to stick that out? The three choices were SmallCo, MidCo and BigCo, with respectively a few hundred million in revenues, a billion+ in revenues and a few billion in revenues. Our system was big enough to materially move the needle for SmallCo, but would have been just nice-to-have, small revenue streams for MidCo and BigCo.

First, we did a rough Return on Investment (ROI) calculation for every supplier. This kind of analysis is very hard to do because each supplier had a very different cost structure. It is one of the hardest to understand categories of financial analysis, trying to understand why companies working on the same product have such divergent cost structures. We did not have access to any of the suppliers’ internal accounting, but we were able to come up with one key statistic – headcount.

SmallCo has about 800 employees, half of whom are engineers supporting all of the company’s R&D work. MidCo had about 150 people assigned solely to this project, plus some corporate resources that may or may not provide additional support. BigCo had 800 engineers just working on this project, plus another 100 SG&A and corporate staff to provide support. To be clear, BigCo had more engineers on this project than SmallCo had total employees, and more engineers than both of the other two combined.

At first blush, it would seem that BigCo was the way to go. They were clearly pouring the most resources into this product. However, this analysis is not that simple. We also had to gauge whether BigCo will stick with that resource commitment. Or will they face investment fatigue and kill the product for insufficient return? Remember, there are two pieces to ROI – the Return and the Investment. No matter how big the Return, the Investment may be too large to generate a meaningful ROI.

We then compared the headcount data with the other data we had which is product quality. I will simplify a very nuanced comparison of these products, there are obviously a lot of technical factors which I will not get into. Put simply, SmallCo had a good-enough product. It was half a generation behind the other two, but would suffice for today’s needs, and they had a roadmap that kept them competitive, at least at the good-enough level. It turns out that MidCo’s product was the best. It had the most advanced features, and performed really well in our benchmarking. BigCo’s product was somewhere in the middle. It had all the advanced, current generation features, but just did perform quite as well as MidCo’s offering.

For us, that was the most surprising part. BigCo’s product, while good, was not the best, despite having so many more resources. Some of this is just a function of that engineering wisdom “nine pregnant women cannot make a baby in one month”. Many engineering problems just need time to sort through. However, there was clearly something else going on. That ‘something’ is the reason we decided to write this post, as we have encountered this problem so many times in tech. Big companies all have their own pathology of underperformance. Intangible factors like company structure and culture can have a big impact on tangibles like delivery data and performance. Every company has their hidden pools of cost. At one company we worked with, it was their oversize operations team. This problem went undetected for years, because the costs of this team were distributed among many P&L line items, and cultural sensitivities made it impossible for anyone to untangle all the threads. But the end result was clear, the company had lower margins than its competitors despite being a generation ahead in technology. Other companies have bloat in other technical functions, while plenty have corporate overhead bloat – too many lawyers and too many contracts, or too many accountants paralyzing decision making

When we investigated BigCo’s cost structure a little further, we found a lot of confusion about the comparison. BigCo management said we were mis-measuring product performance, and that if we did a bunch of other measurements we would see their product was far ahead of the other two. This may be true, but those other measurements had nothing to do with our requirements. Nor do we think any other potential customers value those metrics. Some people suggested that MidCO just had better engineers. Their team was located in an area with lots of component design talent, and so the argument went MidCo was able to pull the best of the best. We had no way to prove or disprove that claim, but with BigCO’s 8:1 headcount advantage, it is hard to imagine engineers who are all 9X better than BigCo’s. Remember these are all successful companies in other fields. Our internal team thought that BigCo was targeting a different market than we operated in, or a different customer pool, so was less concerned about making a product that met our needs. Again, possible but tough to verify. In the end, we never did reach a definitive conclusion on this problem. I highly suspect that the problem was that BigCo had done a bad job of understanding their market. They had not spent enough time listening to customers, building the product they thought we wanted, not the one we actually did. Recall that they had 800+ engineers and far fewer sales and marketing people.

In the end, this did not disqualify them. Because we also had to look at some even more intangible considerations. Notably, MidCo, despite having the (arguably) best product was about to engage on a big cost-cutting initiative. These cuts were well telegraphed by the company’s management. So we became concerned that MidCo would likely cut the entire product line. Even with us as a customer, it would still be a long time before the product was profitable. We remained very concerned that BigCo would go down a similar path. In speaking to senior management at BigCo, they generally gave very positive signals about the product line we cared about. But we also knew that situations can change. But SmallCo was not a clear favorite, they were just so much smaller than the other two, we had to worry about their longevity as well.

In the end, we went with MidCo. The product was the best, and they were willing to sign the best support contract. However, it was not lost on us that neither BigCo nor MidCo would commit to supporting the system as long as would have liked. What confounded our client the most was they realized the biggest difficulty in this whole analysis was judging highly qualitative factors – the management ability of their suppliers. Mismanagement can make MidCos out of BigCos, and can make MidCos and SmallCos targets for acquirers who scramble product lines.