Rating Search Engines By Playing Trivial Pursuit

We’ve written many times about how search engines are moving to presenting direct answers in response to some questions.
Trivia Stump Smart Search Engines from the Associated Press has writer Michael Liedtke deciding to
test how well these direct answers seem to work by involving the search engines in a game of Trivial Pursuit.

Invited to play were Google, Yahoo, MSN, Ask Jeeves and Answers.com. Ask and Answers were a “small step” ahead of the others, but even they rarely met the goal of providing
a single clear answer, he writes.

The scorecard, with points awarded for getting him to the right answer via the first link on the page, with twenty queries in all:

  • Ask & Answers.com: Tied with 10 out of 20 times, 50 percent success rate
  • Google: 8 out of 20, 40 percent success rate
  • Yahoo: 5 out of 20, 25 percent success rate
  • MSN: 2 out of 20, 10 percent success rate

Of course, a search engine might more fairly be deemed successful if it listed a page leading to the right answer anywhere on the search results page, not just in the first
link. Liedtke took that into consideration. He also examined how often they completely failed to lead him to a page with the answer after he scanned a generous 3 pages of
search results. The scorecard:

  • Ask, Answers.com & Google: Failed 3 out of 20 times, 15 percent failure rate (or 85 percent success rate)
  • MSN & Yahoo: Failed 6 out of 20 times, 30 percent failure rate (or 70 percent success rate)

Related reading

Simple Share Buttons