We all love benchmarks, don't we? The definitive way in which we can find out if a piece of hardware is up to snuff.  All those numbers make beautiful charts and graphs and tell the absolute 100% truth about the performance of that hardware in all situations, right? They are an accurate representation of the truth, right? Well. Not really.

In ideal situations, benchmark makers do their best to make sure that their software is able to function equally well on all types of hardware, as to give as clear a picture as possible of the true power of the: CPU, GPU, RAM, HDD, SSD, or anything else that needs to be benchmarked. Ideal situations are, however, rare. And you will see benchmark makers caught red handed colluding with certain hardware vendors in order to make a specific piece of hardware seem better than another one.

One of the more grievous of such cases was Intel colluding with the BAPCo Consortium, the very group created to make fair benchmarks, to undermine AMD CPUs. This isn't some allegation, not some conspiracy, it's what the Federal Trade Commission decided was going on, and since then Intel is obligated by law to specify that benchmarks like SysMark or MobileMark may be optimized for Intel hardware whenever they advertise performance results from those benchmarks. It's a small print, and since the decision was passed down in 2010, people seem to have forgot, so when AMD does try to remind people that Intel attempted to lie to people, they are ridiculed.

AMD got a lot of flack two years ago for posting the video above, being ridiculed left and right by arm-chair computing experts. Generally the same kind of people that are now giving expert crypto advice, by making "bitcoin is worthless now" memes all over the internet. And even though they may seem harmless, they echo enough FUD so that people who aren't familiar with the subject will believe that to be the truth. 

The best way to make sure that there aren't any shenanigans going on in a benchmark is by not relying on closed source ones. As long as the code is obfuscated, as long as documentation is limited, there will always be suspicion. But if it's open, if it's free (free as in libre, not gratis), then there's transparency and with that come trust and confidence that the result will be a reflection of real hardware potential. With no crippling paths made for vendors of products that aren't in bed with the manufacturer of the benchmark. 

That would be another one of those ideal situations. However, there is a workaround that will still give the end user false information that won't reflect reality. This is especially true in the case of GPUs, where the driver can detect when a benchmark is loaded and will use an optimized codepath that can heavily increase performance. Now, if the GPU driver does that with everything, there's no problem, it's a great thing to know that Nvidia or AMD have the time, money and manpower to optimize the driver for each and every game, until it becomes a bloated behemoth weighing in at several GBs. But they don't. There are those game ready drivers that tweak performance, but you won't see it for every game, and you won't see it to the extend that it can be done with a benchmark, because a benchmark is a known quantity. It will always behave in the same exact way, it will always display the same exact sequences, it will always work in the exact same way. A game will not. A game tends to not follow the same exact flow each time... unless we're talking about an in-game benchmark, then yes, it totally will. But the actual game will always have a level of diversity that can't be optimized as well as a canned benchmark. 

For this reason, you shouldn't really put much trust into canned benchmarks. Even if the test itself is made to be as transparent as possible, there's always the chance of someone making shenanigans. And before you go: "Oh, who would ever do that? Cheating on a benchmark is kids stuff, a serious industry wouldn't do that.", need I remind you that VW and others have been doing it in the auto industry for decades, giving people lung cancer in the process.

When you look up the review of a CPU or a GPU, always skip the 3DMarks, skip the SysMarks, skip the PCMarks, skip anything that isn't a direct test on real world a situation. Sure, these test take longer to do, and have to be done multiple times to weed out extremities, but the reviewers that actually go through the trouble of doing it understand that it is the better way of giving the end user an actual clear picture of what the hardware is capable of. Of the performance they can really expect when they get their hands on new components. Otherwise we'd still be living in the world where everyone sings the praises of Pentium 4s and Pentium Ds, only for them to be utter garbage when you actually put them to work. As a former Pentium D owner, I can attest at how trash it actually was.

So always keep your eyes on what benchmarks are being used when you're looking up information about hardware performance. It'll save you a lot of grief and headaches in the years to come.

--------------

For more content made by me, please check out the gaming and hardware videos I make over at @gaminghd and on @free999enigma 's channel.

Also, check out the stuff my friends @stefanonsense @ropname and @vladalexan make, if you like video games.