Where Are My Pages? A Submission Update

Have you submitted pages and not seen them appear in some places? A number of services have been showing oddities of late. Here's a rundown on the latest developments:

At HotBot, new submissions have not been processed for over a month or longer, in some cases. By no means are pages getting listed within the two weeks promised on the page.

Since submitting to HotBot is really submitting to the Inktomi search engine that powers it, this delay also means that new pages are not getting listed in the Inktomi-powered MSN Search, or in the secondary results that Inktomi provides to GoTo, Snap and Yahoo.

Inktomi has added all these new, major partners in the last few months, and the delays are related to that. The service has been concentrating on expanding its ability to serve more users, rather than index new pages.

Pages that have already been listed are still being visited regularly, so the Inktomi index isn't completely stale. The problem is with completely new sites and pages that are being submitted.

Unfortunately, things aren't going to get better until later this month. If there's any good news in all this, webmasters needn't resubmit.

"We've maintained all of the submissions, and they will be crawled in the order of submission. We are not dropping them on the floor," said Troy Toman, Inktomi's director of search services.

Inktomi also says that things should only get better after this, because they are redesigning the Add URL process.

"One of the things we're trying to do is handle the volume that we see," Toman said. "Not only will it come back, but it will come back in a better state than it was before."

Lycos is displaying a similar sluggishness, with some new test sites not appearing nearly two months after submission. No news yet as to any particular problem or when they may be resolved. Existing pages are being revisited, though it doesn't seem that updated information is actually appearing in the index.

At Excite, there's not been a serious change in crawling patterns, but new information that has been posted online and sent out by customer service may have some people confused. Here's the current situation:

Excite asks that you submit only your home page to the Add URL service. For those with their own servers, this means the root page. For those using free web space, this means whatever is the main page of your web site.

I also recommend that you submit one additional page from your site, ideally a site map. The reason for this is in case there is some problem with the Excite spider accessing your home page. This additional page provides an alternative entrance that the Excite spider can use. Kris Carpenter, Excite's director of search services, agrees with this suggestion.

There is no need to submit more pages than this, and there is no real advantage in doing so. Excite does not operate like AltaVista or Infoseek, where submitting a page nearly guarantees it will be listed. The spider will visit your site and try to gather all it can, regardless of whether those pages were submitted.

Should you ignore this advice, be aware that Excite will accept no more than 25 URL submissions from one site in any week. So everything you submit beyond this will not be recorded. This does NOT mean that only 25 pages will be listed from your site. It only means that Excite will not keep a record in its crawl list beyond 25 pages -- and as mentioned, that crawl list is no guarantee that a page will be listed.

Excite says that pages should appear in the index within three weeks. Root URLs are special. These will be indexed and listed usually within a week. Remember, at Excite-owned WebCrawler, this is likely to be the only page listed. So, consider dumping that graphic-intensive splash page for something with search engine friendly text.

Excite also had a bug that was causing it to not index pages hosted on some, not all, virtual servers. In particular, those hosted at WebCom were impacted. This has since been corrected.

Finally, at AltaVista, there are no crawling problems to report, but the service seems to have a bug that is causing many pages to be listed as "No Title" when they have perfectly valid HTML titles. Resubmitting a page may help correct this.