The Search Engine "Perfect Page" Test

How effective are search engines at finding "ideal" search result pages suggested by SearchDay readers? We tested the major engines to find out.

We all have our ideas of what constitutes a "perfect" search result for our favorite queries. In the best of all possible worlds, these "obvious" pages should turn up in the top ten results for their "obvious" search terms.

We asked SearchDay readers to suggest their "ideal" search result pages by sending only URLs, leaving it up to the editors of Search Engine Watch to guess the queries that should cause these pages to make it into the top ten. From dozens of suggestions, we chose 10 web pages to use as test cases.

Suggestions were initially screened for quality and reputable information. We chose some well-known sites, and some relatively obscure pages, to see how the engines handled different types of web content. Then, before we ran the tests, we decided which query terms a "typical" searcher would use to find information related to each page.

Does this test tell you which is the best search engine? No. There are a variety of ways that search engines can be tested. This test, for example, does not try to measure the freshness of results. It doesn't try to examine the overall quality of all the results that came up. A search engine may have found one of our "perfect pages" yet still have nine other poor quality matches.

So what good is this test? We think it gives you a rough idea of how the search engines measure up relevancy-wise. Going forward, we also plan to run further "perfect page" tests along with other methods of rating the search engines. Over time, we hope this will all add up to a battery of measurements that will help you in making your searching choices.

The Engines Tested

We tested the popular or well-known search engines of Ask Jeeves, AllTheWeb, AltaVista, Google, Inktomi, Lycos, MSN Search, and Yahoo. The scores for Google are also relevant to AOL Search and Netscape Search.

HotBot was not tested, because its owner Terra Lycos has said that the search engine is about to undergo a major change in the near future. Thus, we felt it was not worth spending the time to test it now. For the same reason, we also omitted Wisenut, after owner LookSmart informed us that it was also about to undergo a major overhaul. We also opted not to include any meta search engines in this test.

We ran the test on Thursday, October 17th. The results may not be replicable today due to recrawling and reindexing by the search engines that may have changed the relevancy rankings of our selected pages.

The Perfect Page Test Results

Our goal was to evaluate the main or primary editorial results provided by the search engines themselves, so we excluded paid placement listings.

Our scoring was simple -- we awarded one point if an engine returned the "perfect page" in the top ten results for our query. A half-point was awarded for a related page from the same site in the top ten result list. We also awarded half-points for a few unique results.

In one case, Lycos returned a "no results found" message, then later returned the expected page in the number 2 position, so we awarded a half-point to accommodate this glitch. In another case, both Ask Jeeves and AltaVista failed to find the ideal page, but gave the #1 result to a closely related companion web site operated by the same government agency as the ideal page.

Google, MSN Search and Yahoo were the top performers, each with a score of 9.5 out of 10. was next, with a score of 9. Inktomi and Lycos tied, with a score of 8.5. Ask Jeeves scored 8 and AltaVista trailed with a score of 6.5.

Based on these results, we assigned the following letter grades to the engines for the perfect page test:

A Google, Yahoo and MSN Search
B Inktomi and Lycos
B- Ask Jeeves
D AltaVista

In all, the search engines did quite well finding the ideal pages suggested by SearchDay readers. We were pleasantly surprised that there was such a high degree of correlation between the queries that we invented before trying the test searches, and the selected perfect pages themselves.

In rechecking our work, we noticed some significant improvements in Ask Jeeves results -- enough to boost the search engine's score firmly into the "A" range. The Jeeves/Teoma crew had no knowledge of our tests, so the improvements were due to a quiet upgrade that we learned about later. But since all of our scores were calculated from tests run on the same day, we couldn't change Jeeves' score, despite observing the improved results.

Out of curiosity, we also ran our queries at Overture, to see how the test would work for purely paid listings. None of our perfect pages came up in the paid results. Of course, Overture also provides unpaid listings after its paid results. If these are counted, then 4 of our pages made it into the top ten, nonetheless giving Overture what would have been an F grade.

Of course, Overture doesn't really intend for users to perform searches at its site, where there is no attempt to ensure a strong balance between editorial and paid listings. That's the reason we didn't measure it against other search engines on our test scorecard. Overture is really a "search provider," sending its paid listings to search destinations sites such as Yahoo, which blend the paid listings with editorial picks.

In short, while paid listings are useful, the testing in this case shows why a good search engine will want a careful blend of editorial results, as well.

For those of you who have settled on a "favorite" search engine, these results illustrate that other search engines are viable choices. None of the engines consistently returned the target pages as the #1 result.

As mentioned, we plan to run additional tests that are much more rigorous and methodical in the future. We also will publish any comments we receive from the search engines that were included in the test, should they offer any feedback.

Perfect Page Test: Criteria and Detailed Results
A discussion of the criteria we used when selecting engines to test, as well as detailed results for each test query for the perfect page test.

Search Engine Results Chart
Major search engines generally provide listings from a variety of sources, which they may get from third-party search providers or through their own efforts. The table on this page shows where each search engine gets the main results it displays.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Online marketing news
Big names enter war against spam, pop-up ads...
Boston Globe Nov 4 2002 2:44PM GMT
How to trace spammers and deal with them - permanently...
The Register Nov 4 2002 1:45PM GMT
Online access news
Wi-Fi That Follows You Around...
Wired News Nov 4 2002 11:28AM GMT
Online portals news
From masters of garish ads, a portal with Zen...
International Herald Tribune Nov 4 2002 0:07AM GMT
Online search engines news
LookSmart bounces back from the dot-com crash...
Melbourne Age Nov 3 2002 9:40PM GMT
Domain name news
Web Domain Body Approves Controversial Plan... Nov 3 2002 7:02AM GMT
Online search engines news
Skins from AlltheWeb...
Research Buzz Nov 2 2002 5:49PM GMT
Domain name news
The Final Insult: A Lost Domain...
Wired News Nov 2 2002 11:36AM GMT
Online search engines news
FAST Search Engine Unveils New Features, Halloween Skin...
URLwire Nov 2 2002 9:27AM GMT
Domain name news
Three new top level domains?...
Demys Nov 1 2002 5:33PM GMT
Online search engines news
FAST Embraces XHTML, CSS... Nov 1 2002 12:07PM GMT
Domain name news
Country domain rebellion - storm in teacup?...
Demys Nov 1 2002 5:37AM GMT
ICANN Nixes At-Large Reps...
Wired News Oct 31 2002 7:54PM GMT
Online search engines news
Explaining the Recent Yahoo/Google Changes...
Traffick Oct 31 2002 3:33PM GMT
powered by

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was's Web Search Guide.