SEO News
Search

Blocking Crawlers With The Meta Robots Tag

author-default
by , Comments

Most people do not need the meta robots tag on their web pages. Having it on a web page does not help the page get indexed in any way. In fact, the tag is mainly used as a way to keep your web pages out of the search engines. Nor does it even work well for this, since most major search engines do not support it.

Still interested in the tag? The explanations below will explain it in more detail, with an emphasis on clearing up some of the rumors and misconceptions that have grown.

Alternative to robots.txt

The tag was created as an alternative to the robots.txt file, which controls how search engines access a web site (see the Blocking Search Engines With Robots.txt page for more about this) Some people don't have access to a robots.txt file, such as those using free web space provided by an internet access provider. Using the tag can help them keep spiders away, should they desire.

To keep spiders from indexing a web page, use this code:

You need this code on every page you don't want indexed. This is true even if your home page has this code on it:

The code above tells spiders not to index the page (NOINDEX) and not to follow links on the page to other pages in the web site (NOFOLLOW). However, other sites may link to your inside pages. That means spiders may bypass your home page and index the inside pages unless those pages have their own NOINDEX tag.

FYI, this code

Means the same thing as the NOINDEX, NOFOLLOW version shown above.

Some people think this meta tag is important to use:

It tells search engines to index the page and follow links to inside pages, so that they can be index. But search engines will do this even without the code, to the same degree that they index any web site. So don't waste your time with it.

As with all meta tags, this meta tag must go between the tags at the top of your web page.

More Resources

The Web Robots Pages: The Robots META tag
http://www.robotstxt.org/wc/exclusion.html#meta

The official word on using a meta robots tag.


The Original Search Marketing Event is Back!
SES DenverSES Denver (Oct 16) offers an intense day of learning all the critical aspects of search engine optimization (SEO) and paid search advertising (PPC). The mission of SES remains the same as it did from the start - to help you master being found on search engines. Register today!

Recommend this story

comments powered by Disqus