Lycos Transforms Into Directory
In an unprecedented move, Lycos changed itself in April from a crawler-based search engine into a Yahoo-like directory of web sites. Never before has a major crawler-based service abandoned automation in preference to human categorization. Even more dramatic is the fact that Lycos is getting its directory data free, courtesy of competitor Netscape.
To understand the change, a little history is in order. Many readers will recall when I first wrote about NewHoo, a web directory using volunteer editors that launched in June 1998. Netscape acquired NewHoo in November 1998, and the company pledged at the time that anyone would be able to use information from the directory through an open license arrangement. In fact, NewHoo was renamed the Open Directory to underscore the fact that its information was free for reuse. Naturally, Netscape became the first company to do this, using the Open Directory to power a branded version within the Netscape web site.
Why give the Open Directory information away? One reason is that NewHoo had very much a "work for the web" attitude that inspired its volunteers. It's harder to keep them inspired about working for free when the directory is owned by a commercial enterprise. By pledging to make the directory free to anyone, Netscape fueled the impression that volunteers were working for more than just Netscape.
Now its no longer just an impression. Over 10,000 volunteer editors are now powering the main results at Lycos, in an implementation that goes far beyond what Netscape itself has done with the Open Directory information. Three months ago, I quoted Netscape as saying they thought it unlikely that a competitor would use the Open Directory in this way. Now it's happened.
"At the time, every other major portal had their own directory initiative. We didn't think that they would scrap their own efforts to adopt the Open Directory, especially when we were one-quarter of the size we are now. As the Open Directory has grown, gained momentum, and surpassed other smaller directories, it's become more attractive to other portal sites to adopt as a standard platform," said Rich Skrenta, a founder of the Open Directory who now leads engineering on the project.
Unlike some other Netscape portal competitors, Lycos had the big advantage of risking little by making such a transition. Portions of its web index have remained out of date for months. General search engine users may not have noticed this when doing popular queries, because the service has made a point of getting information from selected "premiere sites" into its index on a regular basis and boosting these pages within the search results. But for very specific queries, Lycos had serious freshness problems.
The move to using the Open Directory information neatly pushes those spidering issues aside. Suddenly, Lycos is no longer to be measured against other crawler-based search engines such as AltaVista and Excite. Instead, it is more appropriate to compare it to Yahoo. Crawler-based results are still available, but they are very much secondary to the directory information.
Don't forget that Yahoo remains the most popular search site on the web, standing well above its nearest competitors. Human categorization has given it more relevant results, and users have responded to this. While directories such as Snap and LookSmart are now growing in popularity, they've always lacked Yahoo's first mover advantage. In contrast, Lycos is one of the web's oldest search services. It will be interesting to see if the change to directory results coupled with its already strong brand identity will allow Lycos to close in on Yahoo's popularity.
Lycos isn't just taking the Open Directory and plastering its own logos all over it. The company has excluded some material, such as news archives and encyclopedia data that is not directly related to web sites. More importantly, Lycos is applying its own relevancy mechanisms to sift through Open Directory information and rank the results. Lycos editors are also working to merge Open Directory information with Lycos content.
"I think this is a bold move for us and a very positive one," said Ron Gamble, Director of Directories and Web Guides at Lycos. "We have server tools on our end that really allow our editorial force to integrate high quality Lycos content, and that's really the 'value add' to the users."
Lycos is hoping to go even further into reshaping the Open Directory core into a product of their own creation by introducing a means to rank sites within the directory, instead of relying on the usual alphabetical listings. This should happen in the next few months.
Now for some specifics at Lycos. When you do a search, you'll often be able to access information from up to four main sources: matching categories from the Lycos directory, matching web sites from the Lycos directory, matching articles from news wires and matching web pages that come from crawling the web. A navigational bar that appears near the top of the results pages lets you move between information sources and also identifies which type you are viewing.
For instance, when you do a search for a general topic such as "cars," you'll first be shown matching categories. The word "Categories" in the navigational bar will be highlighted using reverse text, and matching categories themselves come after this, such as "Recreation > Autos". Up to 10 category listings are displayed per page, and you can use the "Next" button to advance to a new page of listings. You can also click on links within the navigational bar to instantly move to matches from one of the other information sources mentioned above, if these are available.
Continuing with our "cars" example, on the next page, the category listings come to an end, which means you are next shown matching web sites. Notice that the words "Web Sites" will be highlighted in the navigational bar above these results.
If you clicked "Next" enough times, you'd eventually come to news information, then finally to matching web pages from the Lycos spider. In reality, most people probably won't find news information by paging through the results. It will either be buried below pages of matching web sites from the directory, or your query will be so specific that there won't be any news items at all. If you want the news information, use the navigational bar to go straight to it.
There isn't always information from all sources. For instance, a search for "tonsils" brings up no categories and only one matching web site from the directory. After this, it's information from the Lycos spider that serves as a backstop, in the same way that Inktomi information serves as secondary results to Yahoo's own directory listings.
There are two other types of information you may see within search results. "First and Fast" material appears at the very top of the first search results page, in response to popular queries. The links here are handpicked as especially relevant to the search topic, such as listing New York City's official web site in response to "New York." Lycos also features its own relevant content such as message boards or travel guides within this area.
"Featured On Lycos" is somewhat similar to "Fast and Fast" in that it features other areas within the Lycos network that might be relevant to your search. This information, if available, appears immediately above the matching web site information or on category pages.
Now let's go back and look more closely at those different information sources, starting with the categories. Remember that car category I cited above? Assuming you click on it, you'll be taken to a page that first displays a variety of subcategories all related to cars. Next, the "Featured On Lycos" section mentioned earlier is displayed. Below this, you'll be shown matching web sites. This format changes slightly as you drill down into the directory. In general, more web sites will be listed rather than subcategories, and you may discover that newsgroups are also listed.
Down at the bottom of the category pages is one of my favorite things about the Open Directory project -- the category editors are listed. You can click on a name and send an editor feedback directly. Its a great benefit for those tired of the idea that editors are kept squirreled away, never to be contacted directly by the public.
When viewing web site pages from the directory, you'll notice a little "Related pages" link that appears at the end of the description. Clicking on this takes you to the category where that site is listed. It's an easy way to see if there are more sites similar to the one you are interested in, especially if your search initially fails to bring up any category matches.
While results from spidering the web have been downplayed, Lycos says it does not intend to cease its web crawling activities.
"The focus on [crawler-based” search definitely is a little bit gone. Nevertheless, we'll have to continue to do a good job on that," Dave Andre, Vice President of Engineering at Lycos. He said the service has just updated its index to eliminate dead links and that it will be scaling up its size in the near future.
Lycos isn't the only one to be making use of Open Directory information. Lycos-owned HotBot also features a branded version, which replaces the LookSmart-powered directory that HotBot used to feature (links to LookSmart content do remain at the bottom of HotBot's directory pages).
To access the directory at HotBot, simply select a top level topic of interest from the home page. Be aware that you aren't shown the entire information available from the Open Directory at HotBot. Listings appear to have been culled to display only the best sites, but I'll have to confirm this in a future issue.
Now that Lycos has made its move, it will be interesting to see if Netscape and its parent AOL decide to embrace the Open Directory as strongly. They have every incentive. Excite continues to power the main search results of Netscape Search and AOL NetFind, but that agreement is apparently being reviewed in the wake of the Excite-@Home deal. Ideally, AOL could use a non-competitive partner to take over for Excite. If Lycos can leverage the Open Directory so dramatically, certainly AOL could do the same.
Such a move would cause the Open Directory to be primarily powering three major search services, and that gives it real credentials for taking the crown away from Yahoo as the web's semi-official phone book. Yahoo has held this position because of the sheer number of people who use it. Webmasters know its crucial they be listed within the guide. But now, it's entirely conceivable that webmasters could begin caring more about being listed within the Open Directory than at Yahoo.
Yahoo still has more than twice the listings of the Open Directory, over 1.2 million versus about 500,000. It also has the advantage of being staffed by professional editors who aren't promoting their own business interests, something the Open Directory has had trouble with in the past. But the Open Directory's editor selection process has tightened up, its listings are showing more breadth, and impassioned volunteers knowledgeable about their topics can build a valuable resource. Even assuming there are a few bad apples, with over 10,000 editors, the Open Directory far exceeds the editing power that Yahoo can apply with its 150 or so editors on staff.
Understanding Lycos Search Results
A simple, illustrated guide explaining the results you see on Lycos.
Lycos Directory FAQ
Explains some of the formatting you'll see as you explore the directory, such as how extremely good sites are shown in bold text.
Lycos to turn ranking system over to users
Boston Globe, April 19, 1999
I like the part that says Netscape has had difficulty convincing other portals to adopt the Open Directory, as if it tried to get Lycos to use Open Directory information, instead of being surprised by the move. The best part is at the end, with the mention of how AOL is facing complaints by volunteers that now say they should get paid for their work. Could the same thing happen with the Open Directory in the future?
Changes in Search Industry Create Strange Bedfellows
New York Times, April 28, 1999
Has comments from other directory sites about the change at Lycos.
The Open Directory
See the service that's powering Lycos and others.
About The Open Directory
Learn how to become an editor here. It's not a free-for-all. Editors are interviewed online, and only 10 percent make the cut, I'm told.
Netscape Integrates Directory
The Search Engine Report, Feb. 2, 1999
A look at how Netscape is using the Open Directory information within its own portal. There are links to other articles I've written about the Open Directory at the end.
Dogpile Open Directory
The metasearch service Dogpile has also added a branded version of the Open Directory to its service.
Much information about the complaint against AOL, provided by some former AOL community leaders. More details about the issue and plenty of links to news articles.