Not Quite A Fair Shopping Comparison Test

Jason Dowell in Comparing Search Engine Results - My Experiment at Search Engine Guide decides to pit shopping search engines against each other. He comes away feeling that Google's Froogle, Yahoo Shopping and MSN are all the same, while Become.com has many more unique URLs. The problem is that he's not comparing like-to-like.

Become isn't currently listing products for sale. It's listing information and reviews about products you may wish to buy. That's great -- a feature we love (and covered more in Chris's Become.com Launches Shopping Search Engine article). But that's not pitting it fairly against the others. To get close to the same type of results on Become, you would have to do ordinary web searches at Google, Yahoo and MSN and then compare. Chances are, the uniqueness would then increase, since they'd be searching against a much wider range of data than just products for sale.

Postscript: Jason notes over here that he was comparing regular search results at the major search engines to Become, not their shopping search results. My apologies for misreading that.

It turns out that my thought that this might give the search engines more unique URLs doesn't pan out, since that's exactly what he did. However, it doesn't change the fact that this still isn't a quite a fair test for other reasons I didn't previously get into, because there was already one big problem I had with the study. But now it makes sense to dive into those other issues more.

Become is hitting a specialized collection of pages -- pages it deems to be shopping related. The major search engines are hitting the entire universe of pages out there. It's still not a like-to-like comparison.

Moreover, if someone is looking for shopping search results, they should be hitting the shopping search services that are offered by the major search engines. That's likely to increase relevancy, if they are in buy mode. Sure, many people don't do this. But they should, and reviewers should be encouraging them to.

If the searcher is in research mode, then Become very much fills a gap the major search engines don't seem to be doing well in. That's because if you're want to find a specialized collection of product reviews, they don't offer this. It's a frustration I've often felt in my own searching, and I applaud what Become is trying to do. But that intent doesn't mean that Become is perfect.

Here's an example. I wanted to buy a color inkjet printer recently. Supposedly, I could do a search at Become and find reviews, rather than people just trying to sell me printers. What do we get for color inkjet printer there?

  1. PC magazine review of one particular HP inkjet, helpful if you understand that you can jump "up" in the site navigation to see other reviews
  2. Lexmark printer reviews -- only they really aren't. It's more a page selling a particular product with four contributed reviews, something you can find on dedicated shopping search engines, but in greater quantity.
  3. Another page from the same site as in number 2 -- in fact, what looks to be a mirror of the page.
  4. Buy.com selling a printer.
  5. What looks to be another mirror on a different site for what was in number 2.
  6. Not a mirror of what was in number two, a different printer, but the same fairly low quality page with a scattered review and seemingly from the same site/data source.
  7. Another shopping page with a single customer review.
  8. A page selling a particular brand of printer, no reviews provided.
  9. Another page from the same site as in 8, no reviews -- in fact, really just a page affiliate linking over to Amazon, where they sell it.
  10. Another page selling a printer, no reviews.

So what did I get from Become? One page I think was great, the first one. Even then, you have to know to jump through the site navigation. Three other pages simply seem to mirror each other. The remaining pages are of low review quality.

Now let's go to Google. I'm only looking at them because it's enough to make my ultimate point.

  1. Page from HP on how to choose a printer. Nice, actually, because it points you to all different models. But not really a review page.
  2. Similar page to the above from HP
  3. Printer reviews page from ConsumerSearch, a good site and great overview
  4. Page for a review of a single, particular printer.
  5. A list of printers for sale by TigerDirect.
  6. Page from about listing Top multifunction printers. Not bad.
  7. HowStuffWorks on how inkjets work. Not really for how to buy, best to buy, but navigating does get me to some shopping search results complete with user reviews.
  8. News.com inkjet review of a particular printer, which with navigation would get me to more.
  9. Amazon selling me an inkjet.
  10. A manufacturer site.

What have we got? One great page and a second, if you know how to jump through the site navigation to other reviews. So double the relevancy of Become. Both Amazon and HowStuffWork also get me to what I suspect are better and more comprehensive user reviews that Become.com.

What's this tell us? That using counts of "unique pages" says nothing about relevancy. The only way to know whether a page is relevant or not, unfortunately, is to actually look at it. For all we know, the unique URLs Become is bringing to the table aren't valuable at all. As just this spot check above shows, three "unique" URLs that Become brought back seem to be mirrors of each other. The fact that Google has fewer unique URLs could indicate that it's actually better at not being duped by mirrors.

Or not -- I'm with Jason that on some queries, Become does feel better if you're looking for shopping reviews. Compare the results for GPS there to Google. Become gives me at least five sources that look (a very quick scan) like excellent beginning points; Google seems to give me one. But again, unique URLs wouldn't tell me any of this. Overall, I love having Become, and I look forward to seeing it evolve.

By the way, for more on the incredible difficulty in measuring search relevancy, see our Coke vs. Pepsi Challenge for Search Engines thread in the Search Engine Watch Forums for a number of links and background I've posted there, along with discussion of the problem.