Problems With Free Web Pages
From The Search Engine Update
Aug. 5, 1997
If you've set up shop on free web pages, you might want to finally make the move to new quarters. It's getting harder to ensure that your site will be indexed, as search engines fight back against spamming attempts coming at them from these venues.
Spammers like free web space, because it's easy to open an account and flood a search engine with multiple submissions. If the address is banned for spamming, they simply move over to a new address. Setting up a new account is a piece of cake.
As a result, some free web space domains are being entirely barred from the search engines. Infoseek is no longer accepting form-based submissions from GeoCities (though submissions can still be done by email). Tripod members have endured similar problems at Alta Vista, as spammers quickly use up the Add URL allowance AltaVista grants each domain. HotBot recently reported to one webmaster that it is missing some free web pages from a variety of sites, including GeoCities, Tripod and America Online.
More alarming is the fact that HotBot may also miss pages from some commercial services that use a common domain, such as WebCom, where a web address might be http://www.webcom.com/˜user/, for example.
The HotBot problem is related to the fact that the search engine's spider won't request more than one page per minute from a server, in order not to overburden the server. Most major search engines request pages "politely" in this fashion. Otherwise, they could bring a server to its knees.
But when a server has thousands, or hundreds of thousands of pages, the polite query method means that it's simply not possible to crawl everything. This problem could be true with some of the other search engines, as well.
HotBot reported that it may be adding an instant Add URL feature to help mitigate the problem.
In short, remember that when you take advantage of free web space, it's as if you live in a house with a lot of roommates. They can get you evicted, even if you did nothing wrong. In the same way, using free web space means that you have a number of virtual roommates, and they might very well cause you trouble in relation to the search engines.
Infoseek Storms GeoCities Pages
Wired News, July 14, 1997