Lycos Partners More Closely With FAST

From The Search Engine Report
July 5, 2000

A longer and more detailed version of this article is available to Search Engine Watch "site subscribers." This is just one of the many benefits that site subscribers receive. Click here to learn more about becoming a site subscriber.

Lycos has finally committed to using FAST Search's results to power the "Web Sites" section of its search results. Over the past two months, Lycos experimented with using results from its own crawler, from Inktomi and from FAST Search in this section. Now, FAST Search, in which Lycos has a significant investment, has been given the nod. The change does not impact Inktomi's deal to power results at Lycos-owned HotBot.

"We'll continue to use both Inktomi and FAST," said Mark Stoever, Vice President of Emerging Destinations at Lycos. In particular, Stoever explained that using different vendors helps distinguish the and sites from each other, thus broadening the Lycos Network's appeal to different audiences.

Significantly, the move makes Lycos the first major search engine to switch from using its own in house web search technology to outsourcing. Lycos started life at Carnegie Mellon University in May 1994 as a crawler-based search engine, using its own spidering system to retrieve pages from across the web. As the Lycos site developed, additional sources of search data were added. However, results from the Lycos spider always were maintained as the dominant information presented.

Then, in April 1999, a landmark move came when the main results began coming from the Lycos version of the Open Directory. The Lycos crawler was retained primarily to serve secondary results, for when Open Directory information was not available. Now, this role has been outsourced to FAST, pretty much taking Lycos out of the do-it-yourself search business, at least on its US site.

The Lycos spider isn't dead. Lycos is retaining it to build specialty collections of content, which will appear in the Lycos results as explained further below. However, the bulk of the burden for providing general web search results now falls to the Open Directory and FAST and to some degree, Direct Hit.

Do a search at Lycos, and the results will start out with a "Popular" section. This is either content from within the Lycos network that editors have selected as relevant to a particular search, or Direct Hit kicks in to provide answers when there is no editor data. You'll know Direct Hit information is being used if the Direct Hit logo appears at the bottom of the search results page. For some very popular queries, Lycos also pops up links in this section to other types of searching you can do with the Lycos network, such as for MP3 files, pictures or homepages. Try a search for "britney spears" to see an example of this.

In the "Web Sites" section, information will come from the Lycos version of the Open Directory, FAST or from the Lycos crawler. From the Open Directory, you might get categories links. (for example, search for "cars," and "Recreation > Models > Scale > Cars" comes up). Click on the category, and you'll see a list of sites for that topic. Additionally, individual sites from the Open Directory might appear in the Web Sites area. These are easy to identify, because they have a link to their "home" category under their listings.

The freshness of the Open Directory information should improve over the next few weeks, as Lycos brings on board a new system. HotBot should be completely in sync with the Open Directory this week, then be updated each week, going forward. The same should happen at Lycos around the end of this month.

If you don't see a category link, then the information is probably coming from FAST Search. You'll certainly see a FAST logo at the bottom of the results page, if any of its information has been used. However, the Lycos crawler may also return some results to this section. This is usually (but not always) information from within the Lycos network. So, look at the URL for the pages listed. If you see "" in them, then that's probably the Lycos crawler, still at work.

The Lycos crawler is also being used to return some results that appear in the "News Articles" section of the results page, with newsfeeds also being employed.

Outside the US, Lycos is still depending on its own crawler to power search results, such as at Lycos Germany and Lycos UK. Also, several of the European Lycos sites have just been upgraded to make use of new directories, which look to be built by volunteers but which are not based on the Open Directory. I hope to take a longer look at this in the near future.

Lycos says it is tapping into the full FAST index, and FAST says that no page that's listed should be more than about for weeks old, with the goal to cut that to only a week, in the future. Additionally, some pages are already revisited and updated more frequently than others, if they change often -- a few even several times per day, FAST says.


FAST Search

FAST has been a major player in pushing up search engine sizes, being the first to break the 200 million page and then the 300 million page mark. The company aims to full-text index 1 billion unique pages later this year.

FAST Search Add URL Page

FAST Natural Language Search

FAST has invested in a natural language interpretation company called Albert, and this beta site lets you have your queries examined for meaning and applied against the FAST index.

FAST Gets Bigger, Partners With Lycos
The Search Engine Report, Feb. 3, 2000

More about FAST Search and how it has previously been powering advanced searches at Lycos.

Lycos to hand off Net-search business
Boston Globe, June 19, 2000

Another take on the Lycos-FAST deal. I have a comment on this being the first big test for FAST, and the company says they've actually been handling a significant load from Lycos for over a month without trouble.

Terra Firma
Boston Globe, June 26, 2000

Nice background on the pending Terra-Lycos merger.

Subtle But Helpful Changes At HotBot
The Search Engine Report, July 5, 2000

Unlike at, Lycos-owned HotBot has no dramatic search provider announcements. HotBot says that it is pleased with its current major providers, Direct Hit and Inktomi, plus says that having a different mix helps the site appeal to a different audience than those using

A longer and more detailed version of this article is available to Search Engine Watch "site subscribers." This is just one of the many benefits that site subscribers receive. Click here to learn more about becoming a site subscriber.