Block search engines and increase ad clickthroughs?

PBS's MediaShift Idea Lab site has an article by Joe Boydston, VP of Technology and New Media at the McNaughton Newspaper Group and a 2009 Knight News Challenge Winner. Boydston runs the websites for three local newspapers that have employed paywalls for some time. In early 2008, McNaughton's management team realized that it was losing ad sales to Google, which was selling AdWords against the terms in articles on McNaughton's own sites. Searchers didn't have to go to the McNaughton sites to get to the articles, and the ad revenues went to Google. So, McNaughton edited its robots.txt files to bar search engines and aggregators from indexing its sites.

Common sense would say that traffic to the McNaughton sites should have gone down, as should have ad viewership, because the sites became "invisible" to Google, Yahoo!, etc. However, every significant metric actually went up (time on site, number of pages viewed per visit, etc.), and most surprisingly, ad clickthrough rates went up almost 10 times over the course of two years.

Boydston believes that the numbers went up because the sites have local news, sports and current events that aren't available from other sources. By denying access to the search engines and aggregators, McNaughton got more people to come directly to its own sites, where they saw more content and engaged more fully with the ads.

This strategy may also make it possible for publishers like McNaughton to run their sites without paywalls. After all, publishers generally impose paywalls because they can't sell enough advertising to be profitable. If they succeed in driving traffic back to their sites by cutting off search engines, their advertising will become more valuable...possibly valuable enough to make the sites themselves free.

The McNaughton strategy will work with sites for which there are no good substitutes with indexed websites. Small- and medium-market local newspapers, as well as high-value specialty publications, fall into that category. National  and major-market newspapers with many competitors probably won't succeed unless everyone decides to block the search engines and aggregators.

In any event, it does represent a possible Achilles' Heel for Google. If more and more publishers decide to block search engines, the value of Google's entire AdWords program will decline. Consider: If Microsoft  had simply paid publishers to block all search engines and aggregators rather than spent hundreds of millions of dollars on Bing, they could have "cut off Google's air supply" and forced Google into expensive revenue-sharing agreements.