Data Mining the Lycos 50

Like many search engines, Lycos offers a glimpse of what's on the world's mind as reflected by user searches. The Lycos 50 is a list of the most popular queries the search engine received over the past week.

A longer, more detailed version of this article is
available to Search Engine Watch members.
Click here to learn more about becoming a member

The list is slightly filtered, to remove general terms, computer terms, company names and sex-related queries. It's also processed so that similar queries are treated as one (for example, spelling variations on "Al-Jazeera").

In addition to the weekly list, Lycos commentator Aaron Schatz pens a daily column describing what search trends are saying about the popularity of various celebrities, organizations, and cultural phenomena.

Schatz's insights are always informative and entertaining. Though his daily reports always use the data from the Lycos 50 as a starting point, his analysis is wide-ranging, thoughtful, and often funny.

"The eclectic nature of the Lycos 50 mirrors the eclectic nature of my personal interests," says Schatz. "I've written freelance articles about local songwriters, comic books, Red Sox slugger Manny Ramirez, Jewish Passover rituals, the history of Boston radio station WBCN, statistical football analysis, and the 2004 Democratic candidates and their relationships to John McCain."

Some daily reports address the most popular terms in a certain category, such as actors, cities, or events, such as the current war in Iraq. Other reports, called "Ones to Watch," look at terms that are rising quickly in search popularity.

As noted in the Lycos 50 FAQ, "these terms are often obscure to the general public, but known by enough Lycos searchers to get picked up by our keen radar. In short they are trends, or future trends, and they often contain the seeds of the 'next big thing.'"

For example, last month Schatz identified a hot medical fad: Seasilver, a vitamin supplement that claims to be a lot more than just a vitamin supplement.

"At this point, I have a pretty good idea of what will be popular and what won't be," says Schatz. "Controversy gets lots of searches. News stories with video attached get lots of searches, particularly if that video isn't shown by the mainstream media."

Schatz also tries to identify likely winners for events like the Oscar and Grammy awards and sports championships.

"I have a general idea of which celebrities will be more popular than others," he says. "Women usually get more searches than men, younger actors more than older actors, and basketball players and female tennis stars more than other athletes not named Tiger Woods."

There's much more than just the daily and weekly lists. Start with the Lycos 50 Elite list, the most popular topics since Lycos began the list in August 1999. Amazingly, there eight topics that have appeared on our list every single week during that time period.

Schatz also compiles Bonus Lists, focused on specific categories. Some of the bonus list categories include celebrities, history holidays sports, and TV and movies.

For webmasters seeking to optimize their sites for search engines, these bonus lists are an excellent -- and free -- resource for keyword research.

Want to look back at formerly popular search terms? The Lycos 50 archives is a complete collection of lists dating back to August 1999.

Schatz also takes an annual look at the unfiltered Lycos 50, with some surprising insights: "We've left in the naughty bits, since they really aren't that naughty," writes Schatz.

What does Schatz observe about users search skills? Are people getting better with their queries that Lycos says average 2.3 words?

"I wish I could say that the answer is "yes" but I don't think it is," he says. You would be surprised how many searches we get for things that nobody in their right mind should be sticking in a search box -- like, for example, 'search'."

"We also get a ton of searches for, and I'm not sure why people aren't just typing those URLs into the address box. My theory is that as people get used to the Internet, they learn not to do things like this, and to search more accurately -- but since the Internet is still growing, we constantly have new users who are still in the early days of learning how this thing works," says Schatz.

The sophistication of search queries also appears to vary based on what people are looking for.

"I've also noticed that different subjects receive different levels of specificity in their searches," says Schatz. "Sports searches, for example, and holiday-related searches seem to be more likely to include multiple words asking for more specific content than searches related to movies or music (except, of course for the ubiquitous 'lyrics.') NASCAR radio frequencies,' for example, or 'Catholic Father's Day sermons.' Searches by the Japanese animation fans are also more often sophisticated," says Schatz.

Though other search engines offer similar glimpses at what users search for (see the "what people search for link, below), in terms of sheer depth, breadth and interpretation, the Lycos 50 stands out as one of the best barometers of what's on the mind of Internet searchers available.

What People Search For
You spent hours fine-tuning your pages to improve your search engine position. Did you make certain to misspell a few of those crucial words? The people searching certainly will. Misspellings, along with X-rated search requests, are common. This page has links to lists of popular search terms from a variety of sources.

You Can Observe a Lot by Watching the Lycos 50
SearchDay, May 8, 2002
"Sultan of Search" Aaron Schatz says you can virtually predict future trends and fads months in advance by watching what people search for on Lycos.

Search Engine Intelligence from the Lycos 50
SearchDay, May 7, 2002
Search Engine Strategies keynote speaker Aaron Schatz entertained and enlightened attendees with the "trends of American culture" he observes as keeper of the Lycos 50 most popular search terms.

A longer, more detailed version of this article is
available to Search Engine Watch members.
Click here to learn more about becoming a member


Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Lycos adds Web-to-phone support to MMS service...
New Media Age Apr 10 2003 12:33PM GMT
Yahoo Revises Estimates Upward, Reports Profit... Apr 10 2003 11:44AM GMT
Iraqi Currency Hot on EBay...
Wired News Apr 10 2003 10:41AM GMT
Google filter blocks innocuous sites...
MSNBC Apr 10 2003 6:02AM GMT
A Government Portal...
line56 Apr 10 2003 5:00AM GMT
In Vintage Maps, a Japan Bygone Floats Lyrically Online...
New York Times Apr 10 2003 1:53AM GMT
Yahoo marketing pays off...
Courier Mail Apr 10 2003 1:50AM GMT
Internet cafi settles copyright suit...
CNET Apr 10 2003 0:01AM GMT
Google turns News 'bug' into payola feature...
The Register Apr 9 2003 10:52PM GMT
War museum launches online database...
Web-User Apr 9 2003 3:33PM GMT
How to Switch Hosting Providers During Your Search Engine Campaign...
Search Engine Guide Apr 9 2003 3:14PM GMT
Iraq, its domain and the 'terrorist-funding' owner...
The Register Apr 9 2003 1:27PM GMT
Countries united by language are divided by web use, says Forrester... Apr 9 2003 9:33AM GMT
powered by