Missing Pages At HotBot
From The Search Engine Update
June 3, 1998
Feedback has come in from a few people confused and worried that their pages seem to have gone missing at HotBot, or that search results have fluctuated over the past weeks. HotBot says this is due to the refreshing its Inktomi-powered index.
The Inktomi index is divided across over 150 machines. Each one holds a portion of the index, and each one dumps everything it holds at least once every three weeks to refresh its listings. On any day, some of these machines are involved in an update.
When updates happen, some pages may be dropped from the index, if they can't be found when that portion of the database is rebuilt. There are various reasons why this happens, such as server problems and network delays. This situation happens with all search engines, but it may be more noticeable with those that refresh their indexes frequently.
The good news is that the situation should rectify itself naturally at the next refresh, when the system tries to retrieve the missing pages again. It will do this because it will have found links to them on other pages it recovered, or because the page may be retained in its crawl list until repeated attempts to find it fail.
You can also speed the process along by resubmitting the page through the HotBot's Add URL form. That form has fluctuated between working as advertised, adding any page within 48 hours, and taking much longer. However, HotBot says Inktomi has made improvements that should make it act more consistently.