IndustryCrawler Submission Chart

Crawler Submission Chart

At-a-glance guide to the submission details for the major crawler-based search engines.

This chart summarizes key details about submitting to the major crawler-based search engines. Explanations can be found below the chart. Specific details about submitting to each particular crawler can be found within the How Search Engines Work section or by clicking on the search engine’s name, in the chart. The last row on the chart shows when information for a particular search engine was last updated.

Search Engine AllTheWeb AltaVista Google Inktomi Teoma
Listing
Tied To Submit?
No Yes No, but may help n/a n/a
Submit Limit No limit No limit No limit n/a n/a
Pages Appear 2-4
weeks
4-6
weeks
About
4 weeks
About
4 weeks
n/a
Overall Freshness 1 day to
4 weeks
1 day to
6 weeks+
1 day to
4 weeks
1 day to
4 weeks
1 day to
4 weeks
Paid Inclusion? Yes Yes No Yes Yes
Inclusion/
Refresh
3 days
2 days
2 days
Daily
n/a 3 days
2 days
7 days
7 days
Cost Per Year $20 first,
˜$15 add
$78 first,
$38-$58 add
n/a $39 first;
$25 add
$30 first;
$18 add
Cost For
1, 10, 100 URLs
$20
$148
$1,431
$78
$600
$3,982
n/a $39
$264
$2,514
$30
$192
$1,812
Self-Serve
Limit
1,000 500 n/a 1,000 1,000
Bulk Program Yes Yes n/a Yes Yes
Powers See Who Powers Whom? Chart
Updated Sept. 15,
2003
Sept. 15,
2003
Sept. 15,
2003
Sept. 15,
2003
Sept. 15,
2003

Listing Tied To Submit?

With some search engines, directly submitting a page greatly increases the odds that the page will be listed. This shows which search engines operate in this manner.

With the others, there is no great advantage in submitting multiple pages from your web site. Instead, they will simply crawl your site and gather what they find of interest independently of your submission attempts.

There is one big exception to this. If you use doorway pages, you will need to submit these individually. Otherwise, the search engine won’t know how to find them. This is because doorway pages typically do not have links pointing at them, for search engines to follow.

Remember, just because you submit doorway pages or any pages directly to a search engine is no guarantee that they will be listed, especially with search engines where listing is not tied to submitting.

Also see the Submitting & Encouraging Crawlers page for more information about the link between submitting and what gets listed in search engines.

Search engines marked N/A have lack a free Add URL page.

Submit Limit

Currently, search engines offering free Add URL pages have no limits on how many submissions they allow per day, per web site. However, keep in mind that the more you submit, the more likely you may attract a review of your site. That’s why for search engines with no limit, a conservative approach would be to stay under 10 pages per day. Also remember that there is usually no reason to submit each page from your web site to search engines where listing is not tied to submitting.

Pages Appear

This shows how soon to expect pages to appear within the listings of a search engine, assuming:

  • They are new
  • They have never been listed before
  • They have not been submitted through a paid inclusion program
  • That a search engine decides to list them at all

Read the tips for each particular search engine in the How Search Engines Work section for advice on increasing the odds that your pages do get selected for listing.

Overall Freshness

This column reflects how out of date a search engine may be for pages that are already listed in the index. For instance, a search engine that has a freshness of 1 day to 4 weeks means it may revisit some pages on a daily basis while others pages may be visited only once every 4 weeks. However, in no case should any page be more than a month out of date. Please note that these figures are those publicly cited by the search engines themselves, in past interviews with them. It is not uncommon for them to have pages more out of date than what they state.

Paid Inclusion

This shows whether a search engine offers a paid inclusion program that will guarantee that your pages are included in its listings.

Inclusion / Refresh

Shows how quickly new pages submitted via paid inclusion will be added to the index and how often they’ll be revisited, after being added.

Cost Per Year

Shows how much paid inclusion costs per URL, on a yearly basis. Prices are based off those from PositionTech for Inktomi & Teoma; Lycos InSite for AllTheWeb and from AltaVista which sells directly.

Cost For 1, 100, 100 URLs

Shows how much it costs to purchase 10, 100 or 500 URLs per year.

Self-Serve Limit

Shows how many URLs you can submit when using “self-serve” style programs that charge flat fees per URL, usually with credit card billing.

Bulk Program

Shows whether there is a “bulk” inclusion program that allows submission of thousands of URLs. Usually, these programs operate on a cost per click basis. In other words, rather than paying a flat fee to be included, you pay a fee each time someone clicks on one of your paid inclusion listings.

Powers

Links to a page that shows some of the major places where listings from the crawler-based search engines will appear, whether this is on their own sites or with important partner web sites.

Updated

Shows when information for a particular search engine was updated on the chart.

Resources

The 2023 B2B Superpowers Index

whitepaper | Analytics The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing

whitepaper | Analytics Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook

whitepaper | Digital Marketing The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

whitepaper | Digital Marketing Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y