IndustryInktomi, Google Win In Recent Relevancy Test

Inktomi, Google Win In Recent Relevancy Test

In December, I called for search engines to get beyond the hype of who is biggest or freshest and develop a commonly-accepted means of measuring actual relevancy. Now, the first such third-part test like this in ages has been done.

In December, my In Search Of The Relevancy Figure article called for search engines to get beyond the hype of who is biggest or freshest and develop a commonly-accepted means of measuring actual relevancy. In it, I wrote of third-party tests that had been commissioned in the past to get at this. Now, the first such third-part test like this in ages has been done.

VeriTest was commissioned by Inktomi to conduct the test. It found that in raw scoring (where URL position wasn’t taken into account), Inktomi came out tops — but just barely. Inktomi earned 1630 points, with Google just behind at 1597. That’s so close that I’d essentially consider the services tied. Behind the leaders came, surprisingly to me, WiseNut at 1277, followed by Teoma at 1275, AltaVista 1222 and AllTheWeb at 1173, another big surprise for coming in last.

Critics will immediately assume that since Inktomi commissioned the test, it would naturally be in Inktomi’s favor. Google itself suggested as much along with others in a recent WebmasterWorld thread.

“It helps a lot to pick the ground rules, what queries to throw out,” Google posted to the thread, pointing out that past tests conducted by the VeriTest on behalf of AltaVista and Ask Jeeves found those services had ranked tops. Not mentioned by Google was that a past test also commissioned by the firm on behalf of Google in September 2000 found — wait for it — Google to be the most relevant.

So is it really just whomever pays for the test gets the best ratings? Not exactly. About two years ago, I moderated a panel involving VeriTest (then known as eTesting Labs). It turned out that some search engines had funded tests where they were NOT found to be the best. In these cases, they didn’t allow the results to be publicly released.

Absolutely, one needs to be critical of any report funded by only one company. All the more reason why I’d hope the search engine industry as a whole would get behind a common set of tests. Let them all pick the “ground rules” and agree that results, favorable or not, will be published for everyone.

As for this particular test, either I or Chris Sherman will likely do a detailed review of it in the near future. But in the meantime, here are a few more details.

There were 100 queries randomly selected from a set of 1 million real ones provided by Inktomi’s search logs. The top 10 editorial results for each of the search engines tested were reviewed. Sponsored listings were not counted. Three judges then reviewed each of the URLs to determine yes or no whether they were “acceptable” relevancywise in relation to the query terms. A raw score as well as two weighted scores based on the position of URLs were then calculated.

From my preliminary review, the main criticism of this method is the “binary” choice of saying whether a document is relevant. Consider a search for “cars.” Any number of pages about cars in some way could be considered relevant. However, how do you know if these are the best documents of the entire set of those possibly relevant? A binary test doesn’t measure this.

To be fair, the test did try to address some nuances of quality. Judges were told to mark only pages they considered using their own judgment to be “excellent” or “good” to be “acceptable” relevancywise, while pages deemed only “fair,” “poor” or other criteria would be rejected. Judges were also told to ask themselves questions such as, “If a friend of mine was interested in the subject of this query, would I email them this URL?,” among others.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication’s search facility, which most have, and search for the headline.

Homeland Security taps privacy official
ZDNet Apr 17 2003 12:35PM GMT
Yahoo Hiring Gateway Shoots for the Stars
SiliconValley.Internet.com Apr 17 2003 10:44AM GMT
Koogle to Leave Yahoo Board; Semel Ascendant
SiliconValley.Internet.com Apr 17 2003 10:44AM GMT
Building a Bigger Search Engine
Wired News Apr 17 2003 9:22AM GMT
Fooling Google
PC Magazine Apr 17 2003 8:50AM GMT
AllTheWeb Adds Dictionary Lookup Feature
BeSpacific Apr 17 2003 5:29AM GMT
Eluding the Web’s Snare
New York Times Apr 17 2003 3:30AM GMT
Honeypot snares raise ethical and legal issues
SiliconValley.com Apr 16 2003 2:27PM GMT
Death knell for Enterprise Portal pure players?
IT-Analysis.com Apr 16 2003 10:41AM GMT
Chinese portals link to challenge Google
ZDNet Apr 16 2003 9:22AM GMT
U.S. Bankruptcy Court Approves EBSCO’s Purchase of RoweCom U.S.A
EContent Apr 16 2003 4:02AM GMT
Being Tops with Your Users and the Search Engines, Part Two
Search Engine Guide Apr 15 2003 2:55PM GMT
Search Engine Robots – How They Work, What They Do (Part I)
Search Engine Guide Apr 15 2003 11:51AM GMT
Overture opens new search services centre in Ireland
Europemedia.net Apr 15 2003 11:36AM GMT
powered by Moreover.com

Resources

The 2023 B2B Superpowers Index

whitepaper | Analytics The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing

whitepaper | Analytics Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook

whitepaper | Digital Marketing The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

whitepaper | Digital Marketing Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y