The search engine features chart below is designed primarily for webmasters who care about how crawler-based search engines index their sites. It provides a summary of important factors and features that can affect how sites are indexed and ranked. Full explanations of items can be found immediately below the comparison chart.
Human-powered search engines like the Open Directory are not listed on this chart because they do not crawl the web to create their listings. See the How Search Engines Work page for an explanation of the differences between crawler-based and human-powered services.
See the Search Engine Features For Searchers page for a summary of how search engines display their results and other information that may be of interest to searchers, rather than search engine marketers and site promoters.
The How Search Engines Work section available to Search Engine Watch members provides more detailed information about the crawlers listed below. This section is just one of the many benefits that Search Engine Watch members receive. To learn more about becoming a member, please visit the membership information page.
This chart covers the crawler of AllTheWeb, AltaVista, Google, Inktomi and Teoma. Some of these crawlers power other search engines, and the relationships are shown on the Search Engine Results page.
|Deep Crawl||AllTheWeb, Google, Inktomi||AltaVista, Teoma|
|Meta Robots Tag||All||n/a|
|Paid Inclusion||All but...|
|Full Body Text||All||n/a||Some stop words may not be indexed|
|Stop Words||AltaVista, |
|Meta Description|| All provide some support, but |
AltaVista, AllTheWeb and Teoma
make most use of the tag
|Meta Keywords||Inktomi, Teoma||AllTheWeb, AltaVista, Google||Teoma support is "unofficial"|
|ALT text||AltaVista, Google, |
All crawlers will find pages to add to their web page indexes, even if those pages have never been submitted to them. However, some crawlers are better than others. This section of the chart shows which search engines are likely to do a "deep crawl" and gather many pages from your web site, even if these pages were never submitted. In general, the larger a search engine's index is, the more likely it will list many pages per site. See the Search Engine Sizes page for the latest index sizes at the major search engines.
This shows which search engines can follow frame links. Those that can't will probably miss listing much of your site. However, even for those that do, having individual frame links indexed can pose problem. Be sure to read the Search Engines And Frames page for tips on overcoming the problems with frames and search engines.
The robots.txt file is a means for webmasters to keep search engines out of their sites. Search Engine Watch members have access to the Blocking Crawlers With Robots.txt page, which covers the robots.txt file in more detail. The Web Robots Pages: The Robots Exclusion Protocolpage also provides official information about robots.txt.
This is a special meta tag that allows site owners to specify that a page shouldn't be indexed. It is explained more on the How HTML Meta Tags Work page and also on the Blocking Crawlers With The Meta Robots Tag page, available to Search Engine Watch members. The Web Robots Pages: The Robots META tagpage also provides official information about robots.txt.
Shows whether a search engine offers a program where you can pay to be guaranteed that your pages will be included in its index. This is NOT the same as paid placement, which guarantees a particular position in relation to a particular search term. The Submitting To Crawlers page provides links to various paid inclusion programs.
All of the major search engines say they index the full visible body text of a page, though some will not index stop words or exclude copy deemed to be spam (explained further below). Google generally does not index past the first 101K of long HTML pages.
Some search engines either leave out words when they index a page or may not search for these words during a query. These stop words are excluded as a way to save storage space or to speed searches.
All the major crawlers support the meta description tag, to some degree. The ones actually named on the chart are very consistent. If you have a meta description tag on your pages, you'll most likely see the content used in some way.
The How HTML Meta Tags Work page explains how to use the meta description tag, and the Search Engine Display Chart for Search Engine Watch members provides a more detailed breakdown of how crawler-based search engines form descriptions.
Shows which search engines support the meta keywords tags, as explained on the How HTML Meta Tags Work page.
This shows which search engines index ALT text associated with images or text in comment tags.
What's New for 2015?
You spoke, we listened! ClickZ Live New York (Mar 30-Apr 1) is back with a brand new streamlined agenda. Don't miss the latest digital marketing tips, tricks and tools that will make you re-think your strategy and revolutionize your marketing campaigns. Super Saver Rates are available now. Register today!