Search Engine Results Continuing to Diverge

A new study suggests that the overlap between search engine results is less than it was even a few months ago, and that the voices of each engine are growing even more unique.

It’s easy to think of search engines as black boxes that all deliver more or less the same results. But that’s not an accurate view, according to a study released today by Dogpile and researchers from the University of Pittsburgh and Penn State University.

The study looked at search results from more than 12,500 random queries on Ask Jeeves, Google, MSN search and Yahoo, and found that the overlap in first page results for these four engines was a scant 1.1% on average for a given query, suggesting that each of the four major search engines has a unique voice that’s not duplicated by the other services.

The overlap between engines increases somewhat when you compare just two or three engines, but it’s still less than the overlap observed in a previous study released last May.

Looking at the organic or natural listings for more than 485,000 first page search results, the study found that:

  • 84.9% of total results are unique to one engine
  • 11.4% of total results were shared by any two engines
  • 2.6% of total results were shared by any three engines
  • 1.1% of total results were shared by any four engines

The implications of these findings are significant for both searchers and marketers. Searchers relying on a single search engine are missing a vast swath of web content that they could easily find simply by trying their queries on other engines. And marketers targeting a single search engine in their optimization efforts are effectively abandoning a potentially huge amount of traffic from other sources.

Just how unique are the results on each engine? On average:

  • 73.9% of Ask Jeeves first page results were unique to Ask Jeeves
  • 71.2% of Yahoo first page results were unique to Yahoo
  • 70.8% of MSN search first page results were unique to MSN search
  • 66.4% of Google first page results were unique to Google

Effectively, for more than two-thirds of all queries, each search engine is likely to give you completely different results. So much for the black box theory, and the idea that all search engines are the same!

These findings were for organic or natural search listings, but the study also found surprisingly little overlap in sponsored listings between Yahoo and Google (Ask gets sponsored listings from Google and MSN from Yahoo). Looking at sponsored listings, the study found that just 4.7% of Yahoo and Google sponsored links overlap for a given query.

Furthermore, for about 15% of all queries, Yahoo did not return a sponsored listing when Google returned one or more, and vice versa. And overall, nearly one-third (29.6%) of all searches lacked a sponsored listing altogether in either Yahoo or Google.

This suggests that opportunities continue to abound for search marketers to gain exposure despite increasing competition among advertisers, particularly for less commonly used “search tail” terms.

Dogpile’s motivation for doing this research is to demonstrate that searchers have a better chance of finding what they’re looking for using a meta search engine There’s merit to this argument, especially with the recent interface changes Dogpile made making it easy to compare the results between the four major engines (Dogpile added MSN search results today).

Beyond a marketing effort, however, the new report goes far beyond the earlier study, presenting a comprehensive summary of methodology, analysis and an excellent discussion of assumptions made and conclusions drawn. For example, the study looked only at the first page of results because the majority of search activity (89.8%) takes place on the first page, according to a log file analysis by Infospace, Dogpile’s owner.

This report also makes a clear distinction between natural or organic search results and sponsored listings, something that was not included in the previous report. On balance, the report provides enough background and raw data to allow you to reach your own conclusions apart from those presented by Dogpile.

It’s an excellent read, and well worth the time spent digesting its findings. Search marketers will find lots of useful nuggets for modifying or adapting campaigns based on actual searcher behavior. And even if your interest in search engines is limited to finding information, the eye-opening statistics on result overlap may encourage you to consider using multiple engines, whether via Dogpile’s meta search engine or simply by visiting multiple sites.

Dogpile has made two versions of the report available for download. Both are PDF files.

Different Engines, Different Results – 30 page PDF white paper

Executive Summary – 1 page PDF

Separately, Dogpile has relaunched its “missing pieces” tool as the Dogpile search comparison tool. This interactive tool lets you enter a query and then displays unique and overlapping results on a Venn-diagram, and tells you exactly how many unique and overlapping results are found on Yahoo, Google and MSN. You can also compare Ask Jeeves results using a drop-down menu at the top of each circle. It’s a fascinating way to test search engine overlap yourself.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication’s search facility, which most have, and search for the headline.

Behavioral Analysis: Is Claria Building the Next Great Search Engine?
Traffick Aug 2 2005 12:29PM GMT
Dogpile adds MSN search
CNET Aug 2 2005 12:26PM GMT
Business Technology: Microsoft Vs. Google: A Rorschach Test
Information Week Aug 2 2005 8:14AM GMT
Yahoo News Adding More Video Content
Search Engine Watch Aug 1 2005 10:35PM GMT
Revisiting Hijacking & Redirects: Moving To A Solution
Search Engine Watch Aug 1 2005 10:32PM GMT
Lucrative Keyword Series: Research Step Two
Search Engine Guide Aug 1 2005 3:20PM GMT
UK Directories: UpMyStreet starts geo-targeted ad campaigns
Netimperative Aug 1 2005 3:18PM GMT
Internet Search Stats Reveal Realty Concerns
E-Commerce Times Aug 1 2005 3:12PM GMT
On hiring binge, Google prowling for additional office space
San Jose Business Journal Aug 1 2005 3:07PM GMT
Search Engine Guide: Research: Obstacles, Opportunities on the Road Ahead Aug 1 2005 2:59PM GMT
In Search of Intelligent Search
Intelligent Enterprise Magazine Aug 1 2005 2:55PM GMT
Podcasts? Google is eating Apple dust
San Jose Mercury News Aug 1 2005 2:53PM GMT
Travel Search: adds rental car service
Norwalk Advocate Aug 1 2005 2:51PM GMT
Q and A With New Shopping Search Engine, Part 1
ClickZ Today Aug 1 2005 3:06AM GMT
Microsoft Denies IE 7 Breaks Yahoo, Google Toolbars
Information Week Jul 31 2005 8:22PM GMT

Related reading

how to increase conversions: ideas, tools, and examples
content formats proven to maximize link acquisition for digital pr
luxury marketing search strategy, part one: consumer mindset
Google's RankBrain: Clearing up myths and misconceptions