The Search Engine "Perfect Page" Test
We all have our ideas of what constitutes a "perfect" search result for our favorite queries. In the best of all possible worlds, these "obvious" pages should turn up in the top ten results for their "obvious" search terms.
We asked SearchDay readers to suggest their "ideal" search result pages by sending only URLs, leaving it up to the editors of Search Engine Watch to guess the queries that should cause these pages to make it into the top ten. From dozens of suggestions, we chose 10 web pages to use as test cases.
Suggestions were initially screened for quality and reputable information. We chose some well-known sites, and some relatively obscure pages, to see how the engines handled different types of web content. Then, before we ran the tests, we decided which query terms a "typical" searcher would use to find information related to each page.
Does this test tell you which is the best search engine? No. There are a variety of ways that search engines can be tested. This test, for example, does not try to measure the freshness of results. It doesn't try to examine the overall quality of all the results that came up. A search engine may have found one of our "perfect pages" yet still have nine other poor quality matches.
So what good is this test? We think it gives you a rough idea of how the search engines measure up relevancy-wise. Going forward, we also plan to run further "perfect page" tests along with other methods of rating the search engines. Over time, we hope this will all add up to a battery of measurements that will help you in making your searching choices.
The Engines Tested
We tested the popular or well-known search engines of Ask Jeeves, AllTheWeb, AltaVista, Google, Inktomi, Lycos, MSN Search, and Yahoo. The scores for Google are also relevant to AOL Search and Netscape Search.
HotBot was not tested, because its owner Terra Lycos has said that the search engine is about to undergo a major change in the near future. Thus, we felt it was not worth spending the time to test it now. For the same reason, we also omitted Wisenut, after owner LookSmart informed us that it was also about to undergo a major overhaul. We also opted not to include any meta search engines in this test.
We ran the test on Thursday, October 17th. The results may not be replicable today due to recrawling and reindexing by the search engines that may have changed the relevancy rankings of our selected pages.
The Perfect Page Test Results
Our goal was to evaluate the main or primary editorial results provided by the search engines themselves, so we excluded paid placement listings.
Our scoring was simple -- we awarded one point if an engine returned the "perfect page" in the top ten results for our query. A half-point was awarded for a related page from the same site in the top ten result list. We also awarded half-points for a few unique results.
In one case, Lycos returned a "no results found" message, then later returned the expected page in the number 2 position, so we awarded a half-point to accommodate this glitch. In another case, both Ask Jeeves and AltaVista failed to find the ideal page, but gave the #1 result to a closely related companion web site operated by the same government agency as the ideal page.
Google, MSN Search and Yahoo were the top performers, each with a score of 9.5 out of 10. AllTheWeb.com was next, with a score of 9. Inktomi and Lycos tied, with a score of 8.5. Ask Jeeves scored 8 and AltaVista trailed with a score of 6.5.
Based on these results, we assigned the following letter grades to the engines for the perfect page test:
A Google, Yahoo and MSN Search
A- AllTheWeb.com
B Inktomi and Lycos
B- Ask Jeeves
D AltaVista
In all, the search engines did quite well finding the ideal pages suggested by SearchDay readers. We were pleasantly surprised that there was such a high degree of correlation between the queries that we invented before trying the test searches, and the selected perfect pages themselves.
In rechecking our work, we noticed some significant improvements in Ask Jeeves results -- enough to boost the search engine's score firmly into the "A" range. The Jeeves/Teoma crew had no knowledge of our tests, so the improvements were due to a quiet upgrade that we learned about later. But since all of our scores were calculated from tests run on the same day, we couldn't change Jeeves' score, despite observing the improved results.
Out of curiosity, we also ran our queries at Overture, to see how the test would work for purely paid listings. None of our perfect pages came up in the paid results. Of course, Overture also provides unpaid listings after its paid results. If these are counted, then 4 of our pages made it into the top ten, nonetheless giving Overture what would have been an F grade.
Of course, Overture doesn't really intend for users to perform searches at its site, where there is no attempt to ensure a strong balance between editorial and paid listings. That's the reason we didn't measure it against other search engines on our test scorecard. Overture is really a "search provider," sending its paid listings to search destinations sites such as Yahoo, which blend the paid listings with editorial picks.
In short, while paid listings are useful, the testing in this case shows why a good search engine will want a careful blend of editorial results, as well.
For those of you who have settled on a "favorite" search engine, these results illustrate that other search engines are viable choices. None of the engines consistently returned the target pages as the #1 result.
As mentioned, we plan to run additional tests that are much more rigorous and methodical in the future. We also will publish any comments we receive from the search engines that were included in the test, should they offer any feedback.
Perfect Page Test: Criteria and Detailed Results
http://www.searchenginewatch.com/searchday/02/pptest-details.html
A discussion of the criteria we used when selecting engines to test, as well as detailed results for each test query for the perfect page test.
Search Engine Results Chart
http://searchenginewatch.com/webmasters/results.html
Major search engines generally provide listings from a variety of sources, which they may get from third-party search providers or through their own efforts. The table on this page shows where each search engine gets the main results it displays.
Search Headlines
NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.
Newsletter signup
Article Archives by Chris Sherman
Farewell, SearchDay! - Dec 29, 2006
Search Engine Forums Spotlight - Dec 22, 2006
Highlights from the SEW Blog: Dec. 18, 2006 - Dec 18, 2006
Search Engine Forums Spotlight - Dec 15, 2006
More article archives
