Articles on robots.txt protocol

RSS

The Taming of the Bots

All savvy search engine marketers know how to create a search-friendly site that can be easily read by a search engine spider, or bot. But what happens when those bots go bad? All savvy search engine marketers know how to create a search-friendly site that can be easily read by a search engine spider, or bot. But what happens when those bots go bad? 0 Comments

No, Google Hasn't Sold Out

Recent changes at Google have provoked cries of protest and alarm from both the media and webmasters. Pay no attention to this hoohaw: Google hasn't sold out. Recent changes at Google have provoked cries of protest and alarm from both the media and webmasters. Pay no attention to this hoohaw: Google hasn't sold out. 0 Comments

Web Spiders List Planned

Search engine spiders and crawlers with other purposes pose significant problems to site owners. In particular, they can skew pageview statistics, which means that advertisers might pay for impressions that human beings never see. Search engine spiders and crawlers with other purposes pose significant problems to site owners. In particular, they can skew pageview statistics, which means that advertisers might pay for impressions that human beings never see. 0 Comments

Google Unveils More of the Invisible Web

Google has quietly extended its index of the web, for the first time making searchable a number of file formats that are all but ignored by other search engines. Google has quietly extended its index of the web, for the first time making searchable a number of file formats that are all but ignored by other search engines. 0 Comments