The Thrill & Danger Of Measuring Relevancy Through Ego Searches

Last week, you may recall that Ask got a rave review from the Wall Street Journal. Robert Scoble saw that, then did an ego search for himself and decided Ask doesn't measure up based on that. Fair criticism? Sure, to some degree. But then again, it's easy to take a single search for anything and show that any of the "leaders" in search have problems, as well.

In Search Of The Relevancy Figure from me back in 2002 goes through the various ways that search engines are measured up. Ego Search is a long-standing one, as I described:

"Ego Search" is another style that can be bad, and one that I still see journalists and others often perform. In an ego search, you look for your name. If you fail to come up tops for it, you conclude the search engine's relevancy is poor.

In some cases, perhaps this is true. If I search for "bill gates," it's reasonable to expect to find the official web site for Microsoft Chairman Bill Gates. But what if you aren't as well known as Bill Gates or have a popular name? What if you've built a web site in some free hosting service that is shared by spammers? These might be issues that push you down, and for good reason. Moreover, are you're going to condemn an entire search engine as bad, based on one search? Well, I've seen it happen.

So is Ask bad because it didn't find Robert Scoble's current blog. Yes, for a variety of reasons:

  • The new blog's been out there enough that I would expect it to be listed in the top results
  • Others can get to it via his old blog, which is listed in the top results, but it would be better not to make that detour
  • It matters to Robert

The last point is probably the most important. While ego searches can be bad, they remain one of the main ways anyone will test their favorite search engine. You've got to get them as right as possible, and that's not happening with Robert.

For the record, in my searching today, Google and Yahoo list both the old and new blog (in that order), while MSN gets the new one first, then the old one further down (though it's the opposite if you search for robert scoble rather than scoble)

Also for the record, Robert himself makes the job tough for any search engine. He does no redirection from his old to new site, which pretty much makes it impossible to do the very best thing, list only the new blog.

Think about it. If you search for Robert, why send you to his old blog, just to make you detour? I'm not saying pages from his own blog shouldn't come up for relevant specific queries. But in a search for him by name, you'd really only want to point people to his current blog.

On the flip side, as I wrote on Dave Naylor's blog recently, keeping both blogs going with no redirection is also smart marketing that ensures you dominate the results for your name :)

Back to the search, I also tried using his first and last name. Ask deserves some kudos on that, as robert scoble puts a picture of Robert at the top of the page, based on pulling material from Wikipedia. Nice! Sure, his new blog still isn't listed, but that doesn't mean some other features aren't helpful.

Earlier I said Ask is bad for failing the ego search test for Robert. But bad as in bad on that specific query, rather than bad overall. Condemning an entire search engine based on it going overboard, as Robert does when he says:

But, Walt, this doesn't portend good things for To be included in the top three you've gotta be as good as Google. Ask isn't even in the same neighborhood yet.

Wow. So all I've got to do is find a query where Ask succeeds and the others fail, and then Ask's in the neighborhood. Well, John Battelle helps out here. The Last Name Test post he just put up shows how he can easily find himself on Ask for his own name, just as he's found on Google, Yahoo and MSN.

Then again, that search also points out an Ask Jeeves failure. I remarked about this on our Daily SearchCast show when talking about Ask's rave review by the Wall Street Journal. Ask was having issues with not consolidating web sites correctly.

In particular, the day of our show, I'd done a search at Ask to find one of my articles. I queried invisible tabs, and I got this:

Searching With Invisible Tabs. By Danny Sullivan, Editor December 2, 2003...
Searching With Invisible Tabs. By Danny Sullivan, Editor-In-Chief December 2, 2003...

That's the same page, listed twice. Ask is having a canonical issue, not realizing that the www URL leads to the same page without the www prefix (and this despite us doing redirection).

You can see the same issue with John's search for battelle:

John Battelle's Searchblog Thoughts on the intersection of search, media, technology, and more. Search and Newsletter

John Battelle's Searchblog

About John Battelle. Federated Media. UC Berkeley. Columns. That Book I Hear He Wrote. Consulting. Contact...

All those listings are from the first page of Ask's results. The first and third listing are the same page, just one is shown with the www prefix.

So now we can say that Ask is clearly not in the same neighborhood as Google, Yahoo and MSN! Hold on there.

Let's go to MSN and look for invisible tabs. I'm not saying that my article should be the only thing listed, but look at the result at the bottom from MSN:

invisible girl
ABchao Brad Bubs Connie Daynah Dodo Dodo's Tabs Heather Jen Jessie Kristin Leah Leigh Sanne Sarah Shiny Shiny Shopgirls! TechLog Meta </a> .org - asm > Valid XHTML XFN WordPress

In this case, I'm getting a page that has nothing to do with invisible tabs of any type other than they simply having both of those words on the page somewhere. But over at Ask, every listing in the first results is related in some way, either to my article, or to how they apply to a programming issue, or to the idea of guitar tabs you use to play the song Invisible. By this measure, MSN isn't in the neighborhood of Ask.

As for domain issues, MSN's fixed the embarrassing issue it had with every subdomain dominating (shall I say domainating) a search for cars, but then when I search for movies today, I get all these from the domain:


None of these are "indented" results, either, so it doesn't feel like the subdomain issue is completely solved. I'm guessing MSN fixed it to check back three levels but not four.

Let's swing back to Yahoo. MSN's domain issues could be excused in part because as a search engine technology, they're still an infant. But here's scoble over at Yahoo showing these domains (numbers show the listing order):







OK, it's not unusual for search engines to show up to two pages from the same domain in the first page of results. Google's popularized this concept through its indented results. I'm not even saying that Yahoo should be indenting results. But if you're going to show more than one page per web site, at least get coordinated enough to put then next to each other. If I just saw a page at the top of my result, I don't want to see another page from that domain at the bottom.

How about the widely acknowledged leader in search, Google? Let's do scoble over there. Down at number six is this:

That's a link-only URL, a page Google only knows about through links pointing at it, rather than having visited the page. What happens if I try to visit it? 404 Page Not Found. Heck, I don't even get that. I get a complete non-resolution to the site, a time-out. It doesn't look to exist any more. So why's this making it at number six for that search? Why isn't Google smart enough to drop this entirely useless listing?

To sum up, ego searches will remain a powerful way anyone measures the relevancy of a search engine personally. However, they also remain a dangerous way to assess the overall health and quality of a service. Ultimately, to measure the quality of a service, you remaining wanting an entire battery of different tests run, measuring relevancy and usability.