Lycos has finally committed to using FAST Search's results to power the "Web Sites" section of its search results. Over the past two months, Lycos experimented with using results from its own crawler, from Inktomi and from FAST Search in this section. Now, FAST Search, in which Lycos has a significant investment, has been given the nod. The change does not impact Inktomi's deal to power results at Lycos-owned HotBot.
"We'll continue to use both Inktomi and FAST," said Mark Stoever, Vice President of Emerging Destinations at Lycos. In particular, Stoever explained that using different vendors helps distinguish the Lycos.com and HotBot.com sites from each other, thus broadening the Lycos Network's appeal to different audiences.
Significantly, the move makes Lycos the first major search engine to switch from using its own in house web search technology to outsourcing. Lycos started life at Carnegie Mellon University in May 1994 as a crawler-based search engine, using its own spidering system to retrieve pages from across the web. As the Lycos site developed, additional sources of search data were added. However, results from the Lycos spider always were maintained as the dominant information presented.
Then, in April 1999, a landmark move came when the main results began coming from the Lycos version of the Open Directory. The Lycos crawler was retained primarily to serve secondary results, for when Open Directory information was not available. Now, this role has been outsourced to FAST, pretty much taking Lycos out of the do-it-yourself search business, at least on its US site.
I can hear some readers asking about other major players that outsource, such as iWon, AOL Search or even HotBot. How is the Lycos action different from what these players do. It isn't, except that those players never had their own internal web search technology to begin with. Lycos did, and its move to outsourcing could mean that we see other players with their own technologies do the same. If it did happen, my guess would be Go (Infoseek) would be next, though the Go's president recently said this would not happen. I suspect Excite might also be an outsourcing candidate, though its recent search improvements do show a continuing commitment to doing search internally.
The Lycos spider isn't dead. Lycos is retaining it to build specialty collections of content, which will appear in the Lycos results as explained further below. However, the bulk of the burden for providing general web search results now falls to the Open Directory and FAST and to some degree, Direct Hit.
Do a search at Lycos, and the results will start out with a "Popular" section. This is either content from within the Lycos network that editors have selected as relevant to a particular search, or Direct Hit kicks in to provide answers when there is no editor data. You'll know Direct Hit information is being used if the Direct Hit logo appears at the bottom of the search results page. For some very popular queries, Lycos also pops up links in this section to other types of searching you can do with the Lycos network, such as for MP3 files, pictures or homepages. Try a search for "britney spears" to see an example of this.
In the "Web Sites" section, information will come from the Lycos version of the Open Directory, FAST or from the Lycos crawler. From the Open Directory, you might get categories links. (for example, search for "cars," and "Recreation > Models > Scale > Cars" comes up). Click on the category, and you'll see a list of sites for that topic. Additionally, individual sites from the Open Directory might appear in the Web Sites area. These are easy to identify, because they have a link to their "home" category under their listings.
The freshness of the Open Directory information should improve over the next few weeks, as Lycos brings on board a new system. HotBot should be completely in sync with the Open Directory this week, then be updated each week, going forward. The same should happen at Lycos around the end of this month.
If you don't see a category link, then the information is probably coming from FAST Search. You'll certainly see a FAST logo at the bottom of the results page, if any of its information has been used. However, the Lycos crawler may also return some results to this section. This is usually (but not always) information from within the Lycos network. So, look at the URL for the pages listed. If you see "lycos.com" in them, then that's probably the Lycos crawler, still at work.
The Lycos crawler is also being used to return some results that appear in the "News Articles" section of the results page, with newsfeeds also being employed.
Outside the US, Lycos is still depending on its own crawler to power search results, such as at Lycos Germany and Lycos UK. Also, several of the European Lycos sites have just been upgraded to make use of new directories, which look to be built by volunteers but which are not based on the Open Directory. I hope to take a longer look at this in the near future.
Lycos says it is tapping into the full FAST index, and FAST says that no page that's listed should be more than about for weeks old, with the goal to cut that to only a week, in the future. Additionally, some pages are already revisited and updated more frequently than others, if they change often -- a few even several times per day, FAST says.
Also, don't expect results at Lycos to match those exactly at FAST. Lycos is using its own porn filtering system, and like any FAST customer, it is also able to tune how FAST returns results, FAST says. Some major options that FAST customers can tweak include the weight to place on link connectivity or popularity, importance of keywords in the title tag, importance on frequency of keywords in the HTML page and keywords in the URL field. Why change setting like these? Imagine someone using FAST on an intranet. Link analysis might be less importance in determining relevance, while title text might be more trustworthy. In contrast, the opposite might be more useful for web results, where link analysis can help balance out webmasters trying to improve placement by creating keyword-rich titles.
Despite the change to FAST, Lycos is maintaining its own Add URL page and recommends that webmasters make use of it. Lycos said it may make use of the Add URL database for its own spidering needs. However, URLs are also supposed to be sent on to FAST Search.
To be on the safe side, I'd recommend submitting your home page and one or two key inside pages to both Lycos and FAST Search, just in case the home page is inaccessible. While technically a double submission, it shouldn't cause you any spam problems, especially as many people who don't know the relationship between Lycos and FAST will be doing this.
Should you deep submit? Both Lycos and FAST say there is no reason, and I would agree. FAST is a very good crawler. Give it a URL to your site, and it should manage to retrieve many other pages. A deep submit really shouldn't be necessary.
"The idea is you submit the top level page you want us to start crawling from, said Knut Magne Risvik, FAST's Research and Development Director of Search Technology. "You can do in depth submission, but we will prioritize the top page of your submit." While FAST has no particular per day URL submission limit, Risvik said that a high rate of submissions, say 10,000 sent quickly for the same site, will cause all of those to be ignored.
Of course, this assumes good linkage between your pages. Any pages that have no external links pointing to them would need to be submitted individual, which is true of any crawler.
By the way, while FAST does ask for a category when you submit using its Add URL page, that information isn't currently being used. As they may make use of it in the future, it makes sense to take a moment and pick the most appropriate category.
FAST has been a major player in pushing up search engine sizes, being the first to break the 200 million page and then the 300 million page mark. The company aims to full-text index 1 billion unique pages later this year.
FAST Search Add URL Page
FAST Natural Language Search
FAST has invested in a natural language interpretation company called Albert, and this beta site lets you have your queries examined for meaning and applied against the FAST index.
FAST Gets Bigger, Partners With Lycos
The Search Engine Update, Feb. 3, 2000
More about FAST Search and how it has previously been powering advanced searches at Lycos.
Subtle But Helpful Changes At HotBot
The Search Engine Report, July 5, 2000
Unlike at Lycos.com, Lycos-owned HotBot has no dramatic search provider announcements. HotBot says that it is pleased with its current major providers, Direct Hit and Inktomi, plus says that having a different mix helps the site appeal to a different audience than those using Lycos.com.
Lycos to hand off Net-search business
Boston Globe, June 19, 2000
Another take on the Lycos-FAST deal. I have a comment on this being the first big test for FAST, and the company says they've actually been handling a significant load from Lycos for over a month without trouble.
Boston Globe, June 26, 2000
Nice background on the pending Terra-Lycos merger.