Amazon's "Pilot Season," and the limits of reviews

Last week, Amazon released a group of television show pilots for audience viewing and feedback. Eight of the pilots are comedies that would (more or less) fit into a 30-minute sitcom slot, while the remaining six are children's shows between 10 and 20 minutes long. Most of the comedy pilots feature involvement by a few well-known entertainment names, including John Goodman, Bill Murray, Bebe Neuwirth, Jeffrey Tambor and Garry Trudeau. However, most of the roles, both in front of and behind the camera, are performed by a mix of experienced but lesser-known talents and complete newcomers.

Amazon's goal is to get "real viewers" to rate the pilots, with the highest-rated pilots getting "greenlighted" for production of an unspecified number of episodes. It's a great idea--instead of a handful of programming executives at a network making the decisions, Amazon has potentially tens of thousands of customers deciding what goes into production. At least, that's the theory.

The reality is that as of this writing, each of the pilots has from a few hundred to a few thousand reviews. In every case, the reviews are skewed heavily toward five stars--which makes it look suspiciously like friends and fans of the creators of the pilots are trying to skew the ratings. With so few reviews, it's difficult to separate the bogus reviews from the real ones, so the decision about which pilots to put into production will ultimately be made by the Amazon Studios programming team--the same people who would make the  decisions at a network.

We know how many people have reviewed the pilots and how many ratings they've given them, but only Amazon knows how many people have viewed each pilot. The actual viewer count is by far the most important measurement, because that can't be gamed--Amazon knows how much of each pilot is viewed each time, so it can subtract out views of, say, less than five minutes (or even use them as a negative indicator--if someone drops out after watching just a few minutes of a video, that's a fairly clear sign that they don't like it.)

My sense is that it may take several "pilot seasons" before Amazon gets enough viewers to be able to make well-informed programming decisions. It'll be interesting to see how Amazon's decisions on this first batch of shows match up with the reviews--if there's any correlation at all.