Meta Search Engine Week!

This week, SearchDay focuses on the world of meta search engines, looking under the hood at how they work and profiling the major players and their offerings.

Meta search engines are powerful tools that search multiple search engines simultaneously. Unlike crawler-based search engines such as Google, AllTheWeb, AltaVista and others, meta search engines generally do not build and maintain their own web indexes. Instead, they use the indexes built by others, aggregating and often post-processing results in unique ways.

When I began planning this roundup of search engines, I thought I'd simply cut to the chase and profile the major players. But I changed my mind after hearing about a study conducted by InfoSpace, the dominant player in the meta search engine space. They found that 86% of the web users they surveyed had no idea what meta search was!

So here is a brief overview of meta search engines. Tomorrow I'll begin the profiles.

Meta search engines accept your query, and send it out to multiple search engines in parallel. They are often quite fast, using private "backdoor" servers made available by the search engines they query. They get this privileged status thanks to revenue sharing agreements.

There are several advantages to using meta search engines, though they're not always the most appropriate tool. The most obvious advantage is that you can get results from multiple search engines without having to visit each in turn. Apart from the time savings, there is some evidence that this gives your search a broader scope, since each individual search engine's index differs from all others.

Personally, I think there are four compelling reasons to use a meta search engine over a crawler-built engine:

* For quick and dirty searches. If you want an answer fast, you may have better luck querying multiple engines simultaneously.

* For broad and shallow searches. Meta searching is an excellent approach if the purpose of your search is to get an overview of a topic.

* To assess potential keywords for an unfamiliar subject. What better way to discover search terms than to see how they appear in a cross section of documents across the web?

* To see how different engines handle the same query. This is an excellent way to get to know the "personalities" of different search engines -- their strengths, weaknesses, and types of queries they handle best.

Meta search engines present results in two ways. One way is to simply list ten or so results from each engine queried with no additional post-processing. Dogpile works this way, listing results from three engines at a time.

Other meta search engines analyze the results and then rank them according to their own rules, combining results from multiple engines into a single, unified list. IxQuick, Metacrawler and Vivisimo are examples of this type of result aggregating meta search engine.

There are some downsides to using meta search engines. Many don't allow advanced search syntax to be sent with your query, so your results may not be as good as when you use the advanced search interface at a specific engine.

And just because they query multiple engines, there can be an illusion of greater coverage than when using a single search engine. This is particularly true when you're searching for popular or commonplace information -- you may end up getting nearly identical results from all queried engines.

Meta search engines also don't solve the "haystack" problem, wonderfully described by Dr. Matthew Koll *. The haystack problem asks "just what are you looking for, anyway?"

  • A known needle in a known haystack
  • A known needle in an unknown haystack
  • An unknown needle in an unknown haystack
  • Any needle in a haystack
  • The sharpest needle in a haystack
  • Most of the sharpest needles in a haystack
  • All the needles in a haystack
  • Affirmation of no needles in the haystack
  • Things like needles in any haystack
  • Let me know whenever a new needle shows up
  • Where are the haystacks?
  • Needles, haystacks -- whatever.

These are just some of the factors to keep in mind when deciding whether to use a meta search engine. All of the meta search engines profiled this week are useful, powerful tools, when used appropriately.

Tomorrow: A look at InfoSpace, the dominant player in the meta search world, and its four different properties.

* Major Trends and Issues in the Information Industry
An overview of information industry issues, including the description of the haystack problem, by Dr. Matthew Koll.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Online search engines news
AlltheWeb's Search Engine Offers New Options...
Research Buzz Sep 16 2002 11:51AM GMT
Domain name news
ICANN delays naming .org successor...
ZDNet Sep 16 2002 11:41AM GMT
Internet features
Store the front page...
Media Guardian Sep 16 2002 10:28AM GMT
Online search engines news
Update: Chinese Google still not itself... Sep 16 2002 9:09AM GMT
Online portals news
New Approach for AOL Broadband...
New York Times Sep 16 2002 4:14AM GMT
Online search engines news
Grading the Search Engines...
Fortune Sep 15 2002 11:32PM GMT
Online portals news
Terry Semel Thinks Yahoo Should Grow Up...
Fortune Sep 15 2002 11:31PM GMT
Web developer news
HTML Tip: More Useful MAILTO Links...
Net Mechanic Sep 15 2002 3:26AM GMT
Domain name news
NetRegistry pushes legal fund to fight domain wars...
ZDNet Sep 14 2002 2:11PM GMT
A PROfile of .pro...
Demys Sep 14 2002 6:06AM GMT
Congress, domain manager clash on kids' Web zone...
ZDNet Sep 13 2002 6:57PM GMT
Online portals news
Yahoo, SBC launch DSL service...
CNET Sep 13 2002 12:52PM GMT
Tech latest
First 'smiley' shows its face...
CNET Sep 13 2002 12:52PM GMT
powered by

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was's Web Search Guide.