Blocking Crawlers With The Meta Robots Tag
Explains how to use this alternative to the robots.txt file method of blocking crawlers.
Explains how to use this alternative to the robots.txt file method of blocking crawlers.
Most people do not need the meta robots tag on their web pages. Having it on a web page does not help the page get indexed in any way. In fact, the tag is mainly used as a way to keep your web pages out of the search engines. Nor does it even work well for this, since most major search engines do not support it.
Still interested in the tag? The explanations below will explain it in more detail, with an emphasis on clearing up some of the rumors and misconceptions that have grown.
The tag was created as an alternative to the robots.txt file, which controls how search engines access a web site (see the Blocking Search Engines With Robots.txt page for more about this) Some people don’t have access to a robots.txt file, such as those using free web space provided by an internet access provider. Using the tag can help them keep spiders away, should they desire.
To keep spiders from indexing a web page, use this code:
You need this code on every page you don’t want indexed. This is true even if your home page has this code on it:
The code above tells spiders not to index the page (NOINDEX) and not to follow links on the page to other pages in the web site (NOFOLLOW). However, other sites may link to your inside pages. That means spiders may bypass your home page and index the inside pages unless those pages have their own NOINDEX tag.
FYI, this code
Means the same thing as the NOINDEX, NOFOLLOW version shown above.
Some people think this meta tag is important to use:
It tells search engines to index the page and follow links to inside pages, so that they can be index. But search engines will do this even without the code, to the same degree that they index any web site. So don’t waste your time with it.
As with all meta tags, this meta tag must go between thetags at the top of your web page.
The Web Robots Pages: The Robots META tag
http://www.robotstxt.org/wc/exclusion.html#meta
The official word on using a meta robots tag.