Lies, Damned Lies, and Benchmarks

We recently saw a post about leaked benchmarks for the new Samsung Applications Processor (AP) that should be an ominous sign for Qualcomm. There is some nuance to the story, but we feel the implications are important. It also give a chance to talk about a peculiar corner of tech analysis – the benchmark.

Some background. Application Processors (AP) are the chips that run most of the software on mobile phones. They are similar in function to the CPU which powers PCs. APs play are a large role in determining the performance of a phone, at least how users perceive that performance. Do apps run smoothly? Are the graphics crisp? Does scrolling move smoothly? How does the phone handle photos? APs do not handle communications with the wireless networks, but they do almost all the other important tasks on the phone. Qualcomm invented the modern applications processor in 2007, and for many years led the pack in AP performance. This is a key strategic product for them, and they developed that market well for many years. True, some companies – notably Samsung – developed their own AP, but for the most part Snapdragon led the pack until recently. Then both Huawei and Apple started to pull ahead with their own AP – Apple’s A Series and Huawei’s Kirin. According to some recent reports, the brand new Snapdragon 888 lags not only the latest Apple A-Series, but actually lags last year’s as well. Not a good look for the company that invented the category. And as we noted elsewhere, is likely part of the motivation for their $1.4 billion acquisition of Nuvia.

Turning to the latest story. This come from Techspot, a site to which we have just been introduced which seems to contain some solid content. They compare data showing the performance of the Exynos 2100, Samsung’s mobile AP, to the recently announced Snapdragon 888. We were not surprised to see that the Exynos is close to the Snapdragon, but lags in some areas. The story then links to “leaked” data on the next version of the Exynos which appeared on a Chinese language website. Obviously, this has to be treated with a grain of salt, but none of the performance claims are particularly outlandish, so seems reasonable.

The conclusion Techspot draws is that Samsung is working very hard to close the gap with Apple and is doing so with a partnership with GPU maker AMD.

There is a lot going on here. A few things we want to highlight.

First, there is this problem with reporting on benchmarks. Marketing departments at chip companies like to play this trick – they compare the current version of their competitor’s products with the future version of their own. Samsung is not the only company to do this (and in fairness, there is no reason to think they are behind the leak), every chip company pulls some version of this trick. Competition among chip makers is usually a steady game of incremental advances. which is what makes Apple’s steady rise to the top all the more dramatic.

Second, our key takeaway from this is the growing sense that competition in the AP landscape is heating up. As we noted in that post on Nuvia a few weeks back, comparing Apple to Qualcomm only tells part of the story. Apple has a performance advantage in that it owns its own silicon and its own software. This allows them to tightly integrate the two and really drive performance advantages. By contrast, Qualcomm has to solve a lowest common denominator problem, weighing features requested by many customers, whose needs have to be balanced. But Qualcomm falling behind Apple is not just a problem for Qualcomm, it is also a problem for Qualcomm’s customers. If the performance is close, the other handset makers can compete with features and distribution. But if the gap gets too wide, their products start to really struggle. This is partially evidenced in Apple’s latest results which show them picking up a lot of share in China. They are filling the void left by Huawei, who has had to depart the market due to US trade restrictions. But it looks a lot like Apple is filling the void, and the other handset makers are not. So we read this latest Techspot post as a sign of the mounting urgency that handset makers feel when competing against Apple while blocked by their chip supplier’s roadmap.

So it should come as no surprise that Samsung, and the others, are working much harder on developing their own APs. This was already happening, but the current situation with Qualcomm has accelerated the shift.

A third takeaway is the importance of AMD in all of this. AMD actually exited mobile a decade ago. They sold their mobile GPU team to Qualcomm, which used the team to build their Adreeno GPU, a big focus of the recent leaks. GPUs (integrated into the AP) matter a lot because they handle image processing, and Apple has made cameras a major competitive feature of smartphones. The fact that AMD has not only re-entered the market, but has become an important player, speaks to missed opportunities at Qualcomm.

Now we do not want to cast this in too negative a light. Qualcomm has immense talent and expertise. They can turn this around. The fact that they would spend $1.4 billion on an acqui-hire, hopefully says they are aware of the problem and rushing to plug the gap. That being said, time is not on their side. The more that Qualcomm’s customers invest in their own APs, the harder it will be for Qualcomm to claw that share back.

Photo by Sarah Kilian on Unsplash

One response to “Lies, Damned Lies, and Benchmarks

  1. Pingback: It’s a Trap – More on Chip Benchmarks | Digits to Dollars·

Leave a Reply