Search Engine Features For Webmasters

The search engine features chart below is designed primarily for webmasters who care about how crawler-based search engines index their sites. It provides a summary of important factors and features that can affect how sites are indexed and ranked. Full explanations of items can be found immediately below the comparison chart.

Human-powered search engines like the Open Directory are not listed on this chart because they do not crawl the web to create their listings. See the How Search Engines Work page for an explanation of the differences between crawler-based and human-powered services.

See the Search Engine Features For Searchers page for a summary of how search engines display their results and other information that may be of interest to searchers, rather than search engine marketers and site promoters.

The How Search Engines Work section available to Search Engine Watch members provides more detailed information about the crawlers listed below. This section is just one of the many benefits that Search Engine Watch members receive. To learn more about becoming a member, please visit the membership information page.

This chart covers the crawler of AllTheWeb, AltaVista, Google, Inktomi and Teoma. Some of these crawlers power other search engines, and the relationships are shown on the Search Engine Results page.

Crawling Yes No Notes
Deep Crawl AllTheWeb, Google, Inktomi AltaVista, Teoma
Frames Support All n/a
robots.txt All n/a
Meta Robots Tag All n/a
Paid Inclusion All but... Google
Full Body Text All n/a Some stop words may not be indexed
Stop Words AltaVista,
Inktomi, Google
FAST Teoma unknown
Meta Description All provide some support, but
AltaVista, AllTheWeb and Teoma
make most use of the tag
Meta Keywords Inktomi, Teoma AllTheWeb, AltaVista, Google Teoma support is "unofficial"
ALT text AltaVista, Google,
AllTheWeb, Inktomi
Comments Inktomi Others

Deep Crawl

All crawlers will find pages to add to their web page indexes, even if those pages have never been submitted to them. However, some crawlers are better than others. This section of the chart shows which search engines are likely to do a "deep crawl" and gather many pages from your web site, even if these pages were never submitted. In general, the larger a search engine's index is, the more likely it will list many pages per site. See the Search Engine Sizes page for the latest index sizes at the major search engines.

Frames Support

This shows which search engines can follow frame links. Those that can't will probably miss listing much of your site. However, even for those that do, having individual frame links indexed can pose problem. Be sure to read the Search Engines And Frames page for tips on overcoming the problems with frames and search engines.


The robots.txt file is a means for webmasters to keep search engines out of their sites. Search Engine Watch members have access to the Blocking Crawlers With Robots.txt page, which covers the robots.txt file in more detail. The Web Robots Pages: The Robots Exclusion Protocolpage also provides official information about robots.txt.

Meta Robots Tag

This is a special meta tag that allows site owners to specify that a page shouldn't be indexed. It is explained more on the How HTML Meta Tags Work page and also on the Blocking Crawlers With The Meta Robots Tag page, available to Search Engine Watch members. The Web Robots Pages: The Robots META tagpage also provides official information about robots.txt.

Paid Inclusion

Shows whether a search engine offers a program where you can pay to be guaranteed that your pages will be included in its index. This is NOT the same as paid placement, which guarantees a particular position in relation to a particular search term. The Submitting To Crawlers page provides links to various paid inclusion programs.

Full Body Text

All of the major search engines say they index the full visible body text of a page, though some will not index stop words or exclude copy deemed to be spam (explained further below). Google generally does not index past the first 101K of long HTML pages.

Stop Words

Some search engines either leave out words when they index a page or may not search for these words during a query. These stop words are excluded as a way to save storage space or to speed searches.

Meta Description

All the major crawlers support the meta description tag, to some degree. The ones actually named on the chart are very consistent. If you have a meta description tag on your pages, you'll most likely see the content used in some way.

The How HTML Meta Tags Work page explains how to use the meta description tag, and the Search Engine Display Chart for Search Engine Watch members provides a more detailed breakdown of how crawler-based search engines form descriptions.

Meta Keywords

Shows which search engines support the meta keywords tags, as explained on the How HTML Meta Tags Work page.

ALT Text / Comments

This shows which search engines index ALT text associated with images or text in comment tags.