Missing Pages At HotBot
From The Search Engine Update
June 3, 1998
Feedback has come in from a few people confused and worried that their pages seem to have gone missing at HotBot, or that search results have fluctuated over the past weeks. HotBot says this is due to the refreshing its Inktomi-powered index.
The Inktomi index is divided across over 150 machines. Each one holds a portion of the index, and each one dumps everything it holds at least once every three weeks to refresh its listings. On any day, some of these machines are involved in an update.
When updates happen, some pages may be dropped from the index, if they can't be found when that portion of the database is rebuilt. There are various reasons why this happens, such as server problems and network delays. This situation happens with all search engines, but it may be more noticeable with those that refresh their indexes frequently.
The good news is that the situation should rectify itself naturally at the next refresh, when the system tries to retrieve the missing pages again. It will do this because it will have found links to them on other pages it recovered, or because the page may be retained in its crawl list until repeated attempts to find it fail.
You can also speed the process along by resubmitting the page through the HotBot's Add URL form. That form has fluctuated between working as advertised, adding any page within 48 hours, and taking much longer. However, HotBot says Inktomi has made improvements that should make it act more consistently.
The Original Search Marketing Event is Back!
SES Denver (Oct 16) offers an intense day of learning all the critical aspects of search engine optimization (SEO) and paid search advertising (PPC). The mission of SES remains the same as it did from the start - to help you master being found on search engines. Early Bird rates available through Sept 12. Register today!