Most people do not need the meta robots tag on their web pages. Having it on a web page does not help the page get indexed in any way. In fact, the tag is mainly used as a way to keep your web pages out of the search engines. Nor does it even work well for this, since most major search engines do not support it.
Still interested in the tag? The explanations below will explain it in more detail, with an emphasis on clearing up some of the rumors and misconceptions that have grown.
Alternative to robots.txt
The tag was created as an alternative to the robots.txt file, which controls how search engines access a web site (see the Blocking Search Engines With Robots.txt page for more about this) Some people don't have access to a robots.txt file, such as those using free web space provided by an internet access provider. Using the tag can help them keep spiders away, should they desire.
To keep spiders from indexing a web page, use this code:
You need this code on every page you don't want indexed. This is true even if your home page has this code on it:
The code above tells spiders not to index the page (NOINDEX) and not to follow links on the page to other pages in the web site (NOFOLLOW). However, other sites may link to your inside pages. That means spiders may bypass your home page and index the inside pages unless those pages have their own NOINDEX tag.
FYI, this code
Means the same thing as the NOINDEX, NOFOLLOW version shown above.
Some people think this meta tag is important to use:
It tells search engines to index the page and follow links to inside pages, so that they can be index. But search engines will do this even without the code, to the same degree that they index any web site. So don't waste your time with it.
As with all meta tags, this meta tag must go between thetags at the top of your web page.
The Web Robots Pages: The Robots META tag
The official word on using a meta robots tag.
Meet Your Favorite Search Engine Watch Contributors
Many of SEW's leading expert contributors will be at ClickZ Live, the new online and digital marketing event kicking off in New York (March 31-April 3). Hear from the likes of: Thom Craver, Josh Braaten, Lisa Barone, Simon Heseltine, Josh McCoy, Lisa Raehsler, Greg Jarboe, Dan Cristo, Joseph Kerschbaum, John Gagnon, Eric Enge and more!