Rate this post

The August issue of Wired Magazine has an article by Chris Colin, titled "Rate This Article", that bemoans the impact of near-universal ratings on the ability of people to make their own decisions. I (mildly) disagree with Colin's belief that ratings interfere with the ability to make serendipitous discoveries--most people would prefer to avoid paying for a bad meal from a bad restaurant, or paying $40 to take their families to a lousy movie.

My concern is that we're increasingly being asked to rate everything, whether or not we have any interest in doing so. If you're an Amazon customer, sooner or later you'll receive an email asking you to rate a purchase. If you use Netflix, it sometimes feels as though you've been asked to rate every movie ever made. If you have your car serviced by a dealership, you'll almost always receive a survey from the manufacturer asking questions about the experience. Before you get the survey, however, your service manager, or whatever title the dealership uses, will implore you to give them the best possible rating on every question.

Most professionals who do surveys for a living, such as market researchers and sociologists, will tell you that there's a big difference between the ratings and reviews that people give without prompting (think Yelp) and those that they're asked to fill out, such as the car dealer's service survey. For most people, they're more likely to write a review or assign a rating for something that they feel strongly about, either positively or negatively. There's no point in giving a "meh" rating.

As for the car service survey, the manufacturer may as well throw the results out. The service manager has already biased your responses (you don't want to get them in trouble), so you're much more likely to give unreasonably high ratings unless something went very wrong. Was the floor of the waiting room so clean that you could eat off of it? Not likely, but you'll still probably give it an "Excellent" rating, so long as you didn't see roaches or rats.

I put more stock in the answers from respondents who choose to rate or review a product themselves, without being prompted to do so. The responses are more emotional, and thus tend to skew to very positive or negative, but they more accurately reflect the true opinions of the respondents at that moment. Note that I'm not including situations where people review products or services in return for compensation; those reviews should be considered biased unless proven otherwise. That's one of the problems with systems that reward reviewers based on the frequency of their reviews, such as Yelp and Amazon. Yelp, for example, throws parties and holds special events for its top reviewers, and Amazon awards special status to its top reviewers. These programs give reviewers incentives to write reviews for products and services that they don't feel strongly about, just to get more reviews posted.

On the whole, the "review economy" that we're living in is a good thing, so long as you consider the source, read the reviews carefully and look out for bias that goes beyond the norm (wild enthusiasm on one side and personal attacks on the other, or reviews that have been influenced by compensation).