Bogus reviews and how to spot them

About a week ago, The New York Times wrote an article about someone who sold bogus positive book reviews that he posted under a variety of identities on Amazon, and presumably, other websites. His company lasted only a few months until it was "outed" by a disgruntled author who didn't like the review of her book that the company posted. The Times used this one company, and one piece of research, to insinuate that there's a torrent of fake book reviews on Amazon, and that all four and five star reviews should be considered to be fake unless proven otherwise.

The Times not only drew a sweeping conclusion from relatively scant evidence, but it also "buried the lede": The problem of fake reviews on the Internet pervades every product category, not just books. It also discounted the fact that fake reviews can be both positive and negative. I chalk up the Times' article to lazy reporting and sloppy editing, but there's a very real problem with fake reviews.

(Update, September 8, 2012: It does appear that Amazon is full of fake book reviews, but according to The Guardian, the practice isn't limited to self-published authors. Author Jeremy Duns figured out that best-selling crime author R.J. Ellory was posting breathtakingly positive reviews of his own books under the pseudonyms "Jelly Bean" and "Nicodemus Jones," and was trashing competing authors with 1-star reviews, including Stuart MacBride and Mark Billingham. Ellory has since apologized, but I suspect that the only thing he's really sorry for was getting caught. The Guardian's article also notes other examples of best-selling authors getting caught giving themselves positive reviews, and in some cases, trashing their competitors. Amazon is going to lose a huge amount of credibility unless it comes up with a way to confirm the identities of reviewers. Publishers could also add clauses to their contracts that prohibit writers from posting reviews under any name other than their own or paying third parties to post reviews.)

It's important to keep in mind that reviews are inherently going to be more heavily distributed toward very positive and very negative ratings, because people are more motivated to review things that they're very happy or unhappy about. Think about going out for dinner to a modestly-priced restaurant and getting an "okay" meal at the price you expected. You're unlikely to write a review about that "meh" experience. On the other hand, what if you get one of the best meals you've ever eaten, or what if the food is bad, the service is worse and the night ends with the waiter spilling a hot cup of coffee in your lap? Either way, you're much more likely to write a review, and the review is likely to be very positive in the former case and very negative in the latter one. So don't automatically assume that great or terrible reviews are fakes.

There's been some research published on how to spot fake reviews. For example, MIT's Technology Review reported that researchers at the State University of New York at Stony Brook used the TripAdvisor site to come up with rules for identifying fake reviews. They started by assembling a group of "likely valid" reviewers--they'd written at least ten reviews, each review was more than a day or two apart, and their ratings didn't deviate too far from the average ratings for all hotels.

The researchers then compared reviews from its "likely valid" group with those of one-time reviewers to see if the one-time reviewers gave a significantly higher number of five-star ratings. They also looked at the ratio of high to low ratings given by different groups of reviewers, as well as sudden bursts of reviews (multiple reviews posted over a few days) that might indicate a deliberate marketing campaign. Then, they compared their results with a previous study they'd conducted, in which they hired people to write fake positive reviews, so that they could identify tell-tale clues such as use of too many superlatives. The researchers found that they could identify fake TripAdvisor reviews "in the wild" around 72% of the time.

The SUNY Stony Brook research focused on fake positive reviews, but a couple of years ago, Consumer Reports' The Consumerist website asked its readers for suggestions on how to spot both positive and negative fake reviews, and they came up with 30 "tells". Here are a few:

• The reviewer only has a single review on the site.
• There's little or no information about the reviewer in their site profile.
• You can't find any information about the reviewer on other sites, such as LinkedIn.
• The reviewer uses a pseudonym that has more than three numbers at the end.
• Multiple reviews, either very positive or very negative, show up about the same subject in a very short period of time (a day, or a few days.)
• The wording of multiple reviews is very similar.
• The review uses the "official" name of the product or service. If it keeps using the official name over and over, it may be an attempt to game search engines.
• There are no details, just a broad statement that the subject is great or terrible.
• They use "marketing speak"--no one would write conversationally the way that the review is written.
• There's a "conversion story"--the reviewer thought that they would hate the product or service, but then they tried it, and now they love it.
• If the subject has multiple locations (such as a chain store or restaurant), the exact same review can be found for multiple locations.
• The review is very negative about the subject and strongly recommends a competitor by name.
• There's a link to the subject's website, or a third party's website, in the review.
• The spelling and grammar in the review is poor--it suggests that the review may have been written by an offshore review mill.

Reviews can save you a lot of money and aggravation, but you have to look for obvious signs of fakery. A fake review can be worse than having no review at all.

Rate this post

The August issue of Wired Magazine has an article by Chris Colin, titled "Rate This Article", that bemoans the impact of near-universal ratings on the ability of people to make their own decisions. I (mildly) disagree with Colin's belief that ratings interfere with the ability to make serendipitous discoveries--most people would prefer to avoid paying for a bad meal from a bad restaurant, or paying $40 to take their families to a lousy movie.

My concern is that we're increasingly being asked to rate everything, whether or not we have any interest in doing so. If you're an Amazon customer, sooner or later you'll receive an email asking you to rate a purchase. If you use Netflix, it sometimes feels as though you've been asked to rate every movie ever made. If you have your car serviced by a dealership, you'll almost always receive a survey from the manufacturer asking questions about the experience. Before you get the survey, however, your service manager, or whatever title the dealership uses, will implore you to give them the best possible rating on every question.

Most professionals who do surveys for a living, such as market researchers and sociologists, will tell you that there's a big difference between the ratings and reviews that people give without prompting (think Yelp) and those that they're asked to fill out, such as the car dealer's service survey. For most people, they're more likely to write a review or assign a rating for something that they feel strongly about, either positively or negatively. There's no point in giving a "meh" rating.

As for the car service survey, the manufacturer may as well throw the results out. The service manager has already biased your responses (you don't want to get them in trouble), so you're much more likely to give unreasonably high ratings unless something went very wrong. Was the floor of the waiting room so clean that you could eat off of it? Not likely, but you'll still probably give it an "Excellent" rating, so long as you didn't see roaches or rats.

I put more stock in the answers from respondents who choose to rate or review a product themselves, without being prompted to do so. The responses are more emotional, and thus tend to skew to very positive or negative, but they more accurately reflect the true opinions of the respondents at that moment. Note that I'm not including situations where people review products or services in return for compensation; those reviews should be considered biased unless proven otherwise. That's one of the problems with systems that reward reviewers based on the frequency of their reviews, such as Yelp and Amazon. Yelp, for example, throws parties and holds special events for its top reviewers, and Amazon awards special status to its top reviewers. These programs give reviewers incentives to write reviews for products and services that they don't feel strongly about, just to get more reviews posted.

On the whole, the "review economy" that we're living in is a good thing, so long as you consider the source, read the reviews carefully and look out for bias that goes beyond the norm (wild enthusiasm on one side and personal attacks on the other, or reviews that have been influenced by compensation).