Several weeks ago I was ruminating about various types of bias and I ran into one recently that I thought to be very interesting in terms of its implications. This is Confirmation Bias.
According to some definition pulled from somewhere “In psychology and cognitive science, confirmation bias (or confirmatory bias) is a tendency to search for or interpret information in a way that confirms one’s preconceptions, leading to statistical errors.”
When I was at MaRS Discovery District, (or should I say, when I was on MaRS) the then VP of Operations, Don Duval used to report that more than 80% of respondents to their survey gave MaRS a 4 or 5 out of five in rating the service they got from MaRS. Now at the time, not thinking about this stat very much, I thought this was an OK result.
The measure on the surface looks like a good one but I have since learned that if your average on customer satisfaction stats is a 4.2 or so, this is actually not a good result. The way I learned was by conducting a workshop where I got an average score of 4.2. I knew that the workshop, while not totally stinking, wasn’t very good. (These things happen.) But the participants were kind enough to give me a 4.2.
At the University of Toronto School of Continuing Studies, where I teach a course on Strategy Execution, my most recent score was a 4.8 and that I consider reasonable but now I don’t find it superb.
The problem is confirmation bias. People who have selected you as a supplier are subject to this confirmation bias as it is a hard thing psychologically for someone to admit they made a mistake. Having invested time in a supplier, a customer will want to confirm their own choice by not rating a supplier as poor. The way they get around this is by scoring someone as 4, not great, just OK. In the parlance of performance evaluations this is probably a “low meets.”
If your survey population is subject to confirmation bias you should be looking for an average above 4.5, not down around 4. And if you’re doing these types of studies, a better methodology to use is Net Promoter Score because it eliminates confirmation bias.