Many people take advantage of free web space provided by their Internet access providers. What they don't realize is that search engines have shown a tendency to miss or even ignore certain sites. Complaints have been heard from those using space provided by America Online, CompuServe and other places.
Indeed, Alta Vista no longer even accepts submissions from Tripod, a popular web service that provides free space. Why? Search engine spammers were using free space there as a base of operations. It's easy to open up a new account, hit the search engine with bogus pages, then move on once the spamming attempt is detected.
That may seem unfair, but when you use free web space, it's as if you have hundreds of roommates. They can get the entire domain in trouble, and the police, or the search engines in this case, may not care that you are innocent.
Ask your provider if there have been any problems with search engines visiting free web pages. They should know if there are complaints, and they should also be able to help resolve any problems. They have the ability to direct large numbers of people toward the search engines, so it's to the advantage of the search engines to work with the providers.
If it's crucial to be indexed, you may want to consider leaving the free web space and going with a commercial hosting service.
CompuServe
The biggest problem CompuServe sites probably have is the instability of the servers. They sometimes go down or are even turned off routinely in early-morning hours. If a search engine tries to visit during these times and gets an error, it may drop a page from it's index.
I've been watching over a four page, CompuServe-hosted site that was established at the beginning of April 1997. Here's the rundown so far, as of 6/12/97:
- Alta Vista: No problem adding all pages via the Add URL form.
- Excite: Pages have yet to appear. Submitted in mid-April, resubmitted in mid May.
- HotBot: Pages have yet to appear. Resubmitted on 6/7/97, with a two-week add promised.
- Infoseek: No problem adding all pages via the Add URL form.
- Lycos: Pages appeared in early June. Submitted in mid-April, resubmitted on 5/18/97.
- WebCrawler: Pages have yet to appear. Submitted on 4/26/97, resubmitted in mid May.
- Yahoo: Has yet to list the site. First submitted on 4/13/97, resubmitted on 5/11/97 and 6/11/97.
Reports From The Field
Here is some information reported by a subscriber Stephen Marsh, who has looked into the issue. I've not yet had a chance to confirm some of this:
- Excite: Reported it will only index up to 50 web pages from AOL every two weeks.
- HotBot: Said that a glitch mean that its Add URL page didn't work for those using AOL's 16-bit browser.
- Lycos: Said that it would not add additional pages from his site beyond the 12 it has already spidered.
Netcom
Reports that Excite will automatically index the home page two weeks after a new web site is created in their free web space. Submission must be done for the other search engines, and there are no notes of problems with having them index pages.
Free Web Pages Indexed
The chart below shows how many web pages have been indexed by the search engines that allow a domain-restricted count. It can give you an idea of the likelihood of being included.
| Site (as of 5/15/97) | AltaVista | HotBot | Infoseek |
| members.aol.com | 652 | 4027 | 89,241 |
| members.tripod.com | 589 | 4813 | 8,857 |
| ourworld.compuserve.com | 7361 | 5459 | 103,525 |
Related Articles
Problems With Free Web Pages
The Search Engine Report, Aug. 5, 1997
http://searchenginewatch.com/_subscribers/articles/article.php/2151231
A further look at some problems those with free web pages may encounter when trying to get listed with search engines.
How Big Are The Search Engines
A look at why pages may not make it into the different search engines.
The Alta Vista Size Controversy
Recounts a public discussion of how Alta Vista only samples some web pages and may completely ignore other sites.
Ethesis Search Engine Summary
http://members.aol.com/Ethesis/search.htm
Stephen Marsh has maintained an AOL-hosted web site since October 1996, and he's tracked how the site has been indexed by the major search engines, over time. Frustrated by the lack of progress, he started up a new, commercially-hosted site and found much better results.
Introducing SES Online
Want to view one of the sessions you missed or listen to an especially informative presenter a second time? SES New York sessions are available for purchase on ClickZ Academy's new e-Learning site. SES is now Online!
