This week, SearchDay focuses on the world of meta search engines, looking under the hood at how they work and profiling the major players and their offerings.
Meta search engines are powerful tools that search multiple search engines simultaneously. Unlike crawler-based search engines such as Google, AllTheWeb, AltaVista and others, meta search engines generally do not build and maintain their own web indexes. Instead, they use the indexes built by others, aggregating and often post-processing results in unique ways.
When I began planning this roundup of search engines, I thought I'd simply cut to the chase and profile the major players. But I changed my mind after hearing about a study conducted by InfoSpace, the dominant player in the meta search engine space. They found that 86% of the web users they surveyed had no idea what meta search was!
So here is a brief overview of meta search engines. Tomorrow I'll begin the profiles.
Meta search engines accept your query, and send it out to multiple search engines in parallel. They are often quite fast, using private "backdoor" servers made available by the search engines they query. They get this privileged status thanks to revenue sharing agreements.
There are several advantages to using meta search engines, though they're not always the most appropriate tool. The most obvious advantage is that you can get results from multiple search engines without having to visit each in turn. Apart from the time savings, there is some evidence that this gives your search a broader scope, since each individual search engine's index differs from all others.
Personally, I think there are four compelling reasons to use a meta search engine over a crawler-built engine:
* For quick and dirty searches. If you want an answer fast, you may have better luck querying multiple engines simultaneously.
* For broad and shallow searches. Meta searching is an excellent approach if the purpose of your search is to get an overview of a topic.
* To assess potential keywords for an unfamiliar subject. What better way to discover search terms than to see how they appear in a cross section of documents across the web?
* To see how different engines handle the same query. This is an excellent way to get to know the "personalities" of different search engines -- their strengths, weaknesses, and types of queries they handle best.
Meta search engines present results in two ways. One way is to simply list ten or so results from each engine queried with no additional post-processing. Dogpile works this way, listing results from three engines at a time.
Other meta search engines analyze the results and then rank them according to their own rules, combining results from multiple engines into a single, unified list. IxQuick, Metacrawler and Vivisimo are examples of this type of result aggregating meta search engine.
There are some downsides to using meta search engines. Many don't allow advanced search syntax to be sent with your query, so your results may not be as good as when you use the advanced search interface at a specific engine.
And just because they query multiple engines, there can be an illusion of greater coverage than when using a single search engine. This is particularly true when you're searching for popular or commonplace information -- you may end up getting nearly identical results from all queried engines.
Meta search engines also don't solve the "haystack" problem, wonderfully described by Dr. Matthew Koll *. The haystack problem asks "just what are you looking for, anyway?"
- A known needle in a known haystack
- A known needle in an unknown haystack
- An unknown needle in an unknown haystack
- Any needle in a haystack
- The sharpest needle in a haystack
- Most of the sharpest needles in a haystack
- All the needles in a haystack
- Affirmation of no needles in the haystack
- Things like needles in any haystack
- Let me know whenever a new needle shows up
- Where are the haystacks?
- Needles, haystacks -- whatever.
These are just some of the factors to keep in mind when deciding whether to use a meta search engine. All of the meta search engines profiled this week are useful, powerful tools, when used appropriately.
Tomorrow: A look at InfoSpace, the dominant player in the meta search world, and its four different properties.
* Major Trends and Issues in the Information Industry
http://www.asidic.org/news/techsumf99.html
An overview of information industry issues, including the description of the haystack problem, by Dr. Matthew Koll.
Search Headlines
NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.
Save up to $400! Register now for SES New York 2012, the Leading Search & Social Marketing Event, taking place March 19-23. Google's Digital Marketing Evangelist Avinash Kaushik will keynote. Hurry, pre-agenda rate expires March 2!
