Articles on robots.txt

RSS

Managing Your Robots.txt File Effectively

Lock

Robots.txt, when used correctly, can help you aid search engines with site crawling. But simple mistakes may stop search engines from crawling your site. Here's how to use robots.txt, and some tools you can use to check for mistakes. While the robots.txt file is a handy tool, it can be dangerous to your SEO efforts if managed improperly. Here's how to use and test the robots.txt file. 0 Comments

Google Webmaster Tools Adds Index Status

google-wmt-index-status-advanced-tab

Google Webmaster Tools will now let you know how many pages of your website are currently included in Google’s index. The new feature can be found by visiting the Health menu. Once there, you’ll find a graph with a year’s worth of data. Google Webmaster Tools will now let you know how many pages of your website are currently included in Google’s index. The new feature can be found by visiting the Health menu. Once there, you’ll find a graph with a year’s worth of data. 0 Comments

Google News Content Now Crawled via Googlebot

google-news

Due to an infrastructure change, Google News is now using the Googlebot instead of a dedicated crawler to examine web content. While most elements remain the same, webmasters should be aware of some minor differences in indexing and crawling. Due to an infrastructure change, Google News is now using the Googlebot instead of a dedicated crawler to examine web content. While most elements remain the same, webmasters should be aware of some minor differences in indexing and crawling. 0 Comments

Beginning Steps

The pages in this section offer background and primer material for those getting started with search marketing, including: The pages in this section offer background and primer material for those getting started with search marketing, including: 0 Comments

Google Adds More Fresh Pages, Changes Robots.txt & 403 Errors, Gains iWon

Over the past month, Google has stepped up the number of pages that it is spidering on a daily basis in an effort to increase the freshness of its database. The search engine has also changed the way it reacts if given a 403 forbidden error message when asking for a robots.txt file and has picked up iWon as a new customer for its search results. Over the past month, Google has stepped up the number of pages that it is spidering on a daily basis in an effort to increase the freshness of its database. The search engine has also changed the way it reacts if given a 403 forbidden error message when asking for a robots.txt file and has picked up iWon as a new customer for its search results. 0 Comments

News Robot Leads To Linking, Indexing Dispute

News search engine News Index crawled The Sunday Times web site, and the paper complained. The situation was resolved peacefully, but the article highlights the possible importance of the robots.txt file where a case might go to court.. News search engine News Index crawled The Sunday Times web site, and the paper complained. The situation was resolved peacefully, but the article highlights the possible importance of the robots.txt file where a case might go to court.. 0 Comments