SEO News

Google PSA: News Publishers Can Use Robots.txt to Block Us

by , Comments

Google is once again reminding news publishers that they do not have to be indexed by the search engine. All they have to do is slap some simple code on a robots.txt file to block the Googlebot.

News publishers, for whatever reason, can't seem to understand that Google doesn't host their content. Perhaps they think that the web is a system of interpipes that are built high in the clouds where they're burning holes through the ozone layer or something. Sigh.

The truth is that news publishers want to charge for access to their sites, just like they charged for print editions. So, they want Google to pay to index their site. If they were truly concerned about the Googlebot, they would simply block it. But they know how much Google sends traffic to their sites. They're just playing dumb.

Remember when newspapers did charge for access to online content? And that didn't work out? So they offered it free with ads? Because they don't own the news and it's going to spread around the interpipes no matter what they charge?

So here's my PSA to news publishers: The web has largely been built on the path of least resistance. And thou protesteth too much.

ClickZ Live San Francisco This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) will bring together the industry's leading online marketing practitioners to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, the comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!

Recommend this story

comments powered by Disqus