Microsoft Quashes Search Relevancy Report

New Search Engine From Microsoft Gets Cool Welcome from the Wall Street Journal (paid reg. required) delves into the rumors we heard in August and confirms them -- that the latest Keynote findings on search relevancy have been suppressed by Microsoft. The last we had were back in January. Why? Well, MSN Search is found to have slipped in those metrics. For its part, Microsoft says it asked for the study not to be released because it questions the methodology.

See? Further evidence it really is 1999 again. I used to get relevancy figures from NPD back then in a scatter-shot approach. I'd know who was ranked first, maybe third, but any search engine that didn't think they were doing well would quash releases of their own figures.

I covered this problem in my In Search Of The Relevancy Figure article in 2002:

Public release of this data is also important. Some companies that contracted with eTesting Labs in the past refused to let the tests be made public, if they did poorly. Similarly, the NPD Group used to do consumer surveys, where the search engines' own users rated their relevancy. Those search engines that did well often released their figures, while those that did poorly kept quiet.

While it may be tempting to sit on bad news, if the search engines want us to take seriously their claims of relevancy, then they have to agree to release both the good and the bad. If a search engine does poorly, then that poor performance should be an excuse to work harder.

I recently dared Google and Yahoo to report on relevancy and get past the size morass. So far, no word. I can tell you that both companies have spent a serious amount of time looking at the best way to spin the issue, far more than I suspect is being spent on coming together to benchmark performance. Guess Microsoft won't be dared at all.

Honestly, if Microsoft and Yahoo execs can collaborate with Google for its stealth Zeitgeist strategic partner meeting, is it really too much to think they can't come together to benchmark relevancy? I know intimately the difficulties in measuring such a subjective thing. But this is your CORE product, gang. This is what you do -- serve search results, and you have no way of proving to the world how good you are. Really, screw knowing who is biggest -- we don't even know which of you is most relevant!

It goes to the business model so crucially. More and more I hear about people talking about Google's search not being as good. Is it? What's the proof of this beyond ego searches? Traffic numbers staying up are one indication it might be doing OK. But the more Google moves away from core search into areas like, hmm, chat and wireless internet access, the more it becomes vulnerable to the "you aren't staying focused" accusations. In turn, those help fuel believe that relevancy might be worse, even if it's not.

In short, even the relevancy winner according to anecdotes and word-of-mouth, Google, has a vested interest showing a relevancy fitness report to the public that can be believed. That's because anecdotes and word-of-mouth can easily turn against it. Google, like all the major players, need to get us the benchmarks.

Meanwhile, I'll say this. Microsoft may be one of Keynote's biggest revenue streams as the WSJ reports and thus has clout to help Keynote make a "business decision" not to release the findings. But here's another business decision to consider. You've just wiped out any faith I have in your figures at all.

The next time Keynote trots them out -- assuming there's a better business climate allowing that to happen -- I'm going to be sitting here dubious if I should even trust them.

Honestly, I can't say enough how tired I am -- and how tired everyone should be -- of these types of games. Google's growing its index -- fact -- but won't budge the home page number until it obviously feels it will be able to wallop any counterclaims by Yahoo. So that number on the home page is meaningless other than as a PR hamburger count stunt. Yahoo played the game as well, of course, deciding it was prudent to finally publish its figures only when knew they'd have a count far larger than Google could trot out. Microsoft's playing the "don't release relevancy figures" game.

We did this once already in Web 1.0 or Search 1.0 or whatever you want to call it. If we're in Web 2.0 or Search 2.0, let's see things advance. No more games. Come together and go fund someone to properly study and benchmark how you are doing, and publicly pledge you will live and die by those results. Don't roll back into the rut of last decade.

Aside from the relevancy quashing, the story also looks at how Microsoft doesn't seem to be gaining in the search wars despite the technological investment in build its own search engine and backing it with marketing money. It has been noticeable as when I did the various traffic numbers last month how MSN hasn't really gained share. One interesting tidbit also is Microsoft thinking of sharing ad revenue with computer makers to help lock in potential search traffic.