Are popular benchmarks valid comparisons of architectures?

Question

Recently, people have been very excited over Apple's ARM processor capabilities, citing benchmarks results that rival x86 processors. However, I am kind of skeptical of these benchmarks: though the two architectures are merging in terms of instruction set size, x86 seems largely CISC while ARM seems largely RISC. I am also skeptical because the power consumption properties of the two architectures are very different, it wouldn't make sense for x86 processors to be about the same, performance-wise, as ARM architectures; otherwise, people would just save money by running ARM processors instead.

So, are these benchmarks legitimate in terms of testing hardware properly? Are there things being untested by these benchmarks that would put x86 ahead of ARM?

Answer 1

"Real world" benchmarks are testing a subset of features that are aligned to program operations rather than raw processor power. It is likely that they are leaving out important tests that favour brute strength and computational power that would show the x86_64 for what it is, a more capable and powerful core.

ARM cores have gotten to the point where they have great memory bandwidth, great bandwidth to storage devices and pretty good graphics cores. What this means is that they have everything needed to perform well in "light" tasks like web browsing and gaming and "office" related tasks.

This is what the more modern "benchmarks" are looking at, a set of tasks that are more user-centric rather than focused on the hardware itself. Yes it does take a certain amount of processing power to load a webpage, but a lot of the task is held up in transferring the images from memory and pushing them to the graphics engine.

Browser JavaScript engines can be better optimised for ARM, leaving out unnecessary features, and making them appear faster as well.

This pattern though can make an ARM processor look as fast as a more powerful processor, because for those tasks it is.

What the benchmarks ignore though is the more intensive tasks we expect to perform on an x86_64 processor. I would expect an ARM core to perform video transcoding, massive physics computations (think big RTS games) or raytraced 3D rendering (blender, Maya) appreciably slower than an x86 core, but those are not necessarily "common" user tasks and so don't get benchmarked.

Essentially for data movement and small amounts of processing an ARM processor could be as fast as an x86 processor. For actual raw power and bulk data processing the x86 will win every time.