Everything is “good” according to NotebookCheck. They’ve come across devices with glaring problems, but even those get a high final score.
How are consumers supposed to distinguish between better and worse when three-quarters of all scores are between 80 and 90?
Here’s how NotebookCheck categorizes the scores:
00 – 49 % – Insufficient
50 – 62 % – Sufficient
63 – 74 % – Satisfactory
75 – 87 % – Good
88 – 100 % – Excellent
This means not a single one of the devices they reviewed in 2018 was unsatisfactory.
Don’t be so easily satisfied, NotebookCheck!
The Problem
Currently, NotebookCheck uses the Weighted Sum Model:
total score = weight × score₁ + weight × score₂ + …
where each scoreₙ is a percentage (decimal from 0 to 1).
It is severely flawed for this purpose. Suppose a ‘Multimedia’ laptop has an abysmal score in one category, let’s say ‘Temperature’ — it severely overheats. That would be a deal-breaker for most consumers in real life, but even if that score were 0, the device still has the chance to get a 92/100 overall rating!
A Solution
The Weight Product Model much more accurately predicts how consumers would consider a product:
total score = score₁ᵂᴱᴵᴳᴴᵀ × score₂ᵂᴱᴵᴳᴴᵀ × …
where each scoreₙ is a percentage (decimal from 0 to 1).
A horrendous sub-score now has the power to impact the overall score in the way a deal-breaker does.
Don’t get me wrong, NotebookCheck is still the best review website for portable computing devices. I commend them for using quantitative measurements and not just qualitative opinions. However, I hope they adjust how they score their reviews, because the current method really isn’t much help.
Hi,
Thanks for the post. Can we have the raw dataset you used for your plots?
Hi there, My name is Simon, Editorial Director of NBC. Interesting analysis, thanks for that. But you forgot one quite important thing: Notebookcheck does not select its review samples randomly, but specifically. Of course we primarily try to review devices that we consider to be particularly exciting for many users.… Read more »