The Thrill & Danger Of Measuring Relevancy Through Ego Searches

Last week, you may
recall that
Ask got a rave review from the Wall Street Journal. Robert Scoble saw that, then
did an ego search for himself and
decided
Ask doesn’t measure up based on that. Fair criticism? Sure, to some degree. But
then again, it’s easy to take a single search for anything and show that any of
the "leaders" in search have problems, as well.

In Search
Of The Relevancy Figure
from me back in 2002 goes through the various ways
that search engines are measured up. Ego Search is a long-standing one, as I
described:

"Ego Search" is another style that can be bad, and one that I still see
journalists and others often perform. In an ego search, you look for your
name. If you fail to come up tops for it, you conclude the search engine’s
relevancy is poor.

In some cases, perhaps this is true. If I search for "bill gates," it’s
reasonable to expect to find the official web site for Microsoft Chairman Bill
Gates. But what if you aren’t as well known as Bill Gates or have a popular
name? What if you’ve built a web site in some free hosting service that is
shared by spammers? These might be issues that push you down, and for good
reason. Moreover, are you’re going to condemn an entire search engine as bad,
based on one search? Well, I’ve seen it happen.

So is Ask bad because it didn’t find Robert Scoble’s current blog. Yes, for a
variety of reasons:

  • The new blog’s been out
    there enough that I would expect it to be listed in the top results
     
  • Others can get to it via his old blog,
    which is listed in the top results, but it would be better not to make that
    detour
     
  • It matters to Robert

The last point is probably the most important. While ego searches can be bad,
they remain one of the main ways anyone will test their favorite search engine.
You’ve got to get them as right as possible, and that’s not happening with
Robert.

For the record, in my searching today, Google and Yahoo list both the old and
new blog (in that order), while MSN gets the new one first, then the old one
further down (though it’s the opposite if you search for
robert
scoble
rather than
scoble)

Also for the record, Robert himself makes the job tough for any search
engine. He does no redirection from his old to new site, which pretty much makes
it impossible to do the very best thing, list only the new blog.

Think about it. If you search for Robert, why send you to his old blog, just
to make you detour? I’m not saying pages from his own blog shouldn’t come up for
relevant specific queries. But in a search for him by name, you’d really only
want to point people to his current blog.

On the flip side, as I

wrote
on Dave Naylor’s blog recently, keeping both blogs going with no
redirection is also smart marketing that ensures you dominate the results for
your name :)

Back to the search, I also tried using his first and last name. Ask deserves
some kudos on that, as
robert scoble
puts a picture of Robert at the top of the page, based on pulling material from
Wikipedia. Nice! Sure, his new blog still isn’t listed, but that doesn’t mean
some other features aren’t helpful.

Earlier I said Ask is bad for failing the ego search test for Robert. But bad
as in bad on that specific query, rather than bad overall. Condemning an entire
search engine based on it going overboard, as Robert does when he says:

But, Walt, this doesn’t portend good things for Ask.com. To be included in
the top three you’ve gotta be as good as Google. Ask isn’t even in the same
neighborhood yet.

Wow. So all I’ve got to do is find a query where Ask succeeds and the others
fail, and then Ask’s in the neighborhood. Well, John Battelle helps out here.
The Last Name Test
post he just put up shows how he can easily find himself on Ask for his own
name, just as he’s found on Google, Yahoo and MSN.

Then again, that search also points out an Ask Jeeves failure. I remarked
about this on our
Daily SearchCast
show
when talking
about Ask’s rave review by the Wall Street Journal. Ask was having issues with
not consolidating web sites correctly.

In particular, the day of our show, I’d done a search at Ask to find one of
my articles. I queried
invisible tabs,
and I got this:

Searching With Invisible Tabs. By Danny Sullivan, Editor December 2,
2003…

searchenginewatch.com/searchday/article.php/3115131

 

Searching With Invisible Tabs. By Danny Sullivan, Editor-In-Chief December
2, 2003…

www.searchenginewatch.com/searchday/article.php/3115131

That’s the same page, listed twice. Ask is having a
canonical issue,
not realizing that the www URL leads to the same page without the www prefix
(and this despite us doing redirection).

You can see the same issue with John’s search for
battelle
:

John Battelle’s Searchblog Thoughts on the intersection of search, media,
technology, and more. Search and Newsletter
battellemedia.com/


John Battelle’s Searchblog

About John Battelle. Federated Media. UC Berkeley. Columns. That Book I
Hear He Wrote. Consulting. Contact…
www.battellemedia.com/

All those listings are from the first page of Ask’s results. The first and
third listing are the same page, just one is shown with the www prefix.

So now we can say that Ask is clearly not in the same neighborhood as Google,
Yahoo and MSN! Hold on there.

Let’s go to MSN and look for

invisible tabs
. I’m not saying that my article should be the only thing
listed, but look at the result at the bottom from MSN:


invisible
girl

ABchao Brad Bubs Connie Daynah Dodo Dodo’s
Tabs
Heather Jen Jessie Kristin Leah Leigh Sanne Sarah Shiny Shiny
Shopgirls! TechLog Meta </a> .org – asm > Valid XHTML XFN WordPress

In this case, I’m getting a page that has nothing to do with invisible tabs
of any type other than they simply having both of those words on the page
somewhere. But over at Ask, every listing in the first results is related in
some way, either to my article, or to how they apply to a programming issue, or
to the idea of guitar tabs you use to play the song Invisible. By this measure,
MSN isn’t in the neighborhood of Ask.

As for domain issues, MSN’s fixed the
embarrassing issue it had
with every cars.com subdomain dominating (shall I say domainating) a search for
cars, but then when I search for
movies
today, I get all these from the go.com domain:

  • movies.go.com
  • casanova.movies.go.com
  • video.movies.go.com/sincity

None of these are "indented" results, either, so it doesn’t feel like the
subdomain issue is completely solved. I’m guessing MSN fixed it to check back
three levels but not four.

Let’s swing back to Yahoo. MSN’s domain issues could be excused in part
because as a search engine technology, they’re still an infant. But here’s
scoble over at Yahoo
showing these domains (numbers show the listing order):

2)
scobleizer.wordpress.com

3) www.kunal.org/scoble

5)
www.kunal.org/scoble/archives/2004_09.html

6)
radio.weblogs.com/0001011/2005/02/19.htm

8)
radio.weblogs.com/0001011/2004/01/27.html

9)
scobleizer.wordpress.com/2006/02/13/the-brrreeeport-report

OK, it’s not unusual for search engines to show up to two pages from the same
domain in the first page of results. Google’s popularized this concept through
its indented results.
I’m not even saying that Yahoo should be indenting results. But if you’re going
to show more than one page per web site, at least get coordinated enough to put
then next to each other. If I just saw a scobleizer.wordpress.com page at the
top of my result, I don’t want to see another page from that domain at the
bottom.

How about the widely acknowledged leader in search, Google? Let’s do
scoble over there. Down at
number six is this:


scobleizer.manilasites.com/

That’s a link-only URL, a page Google only knows about through links pointing
at it, rather than having visited the page. What happens if I try to visit it?
404 Page Not Found. Heck, I don’t even get that. I get a complete non-resolution
to the site, a time-out. It doesn’t look to exist any more. So why’s this making
it at number six for that search? Why isn’t Google smart enough to drop this
entirely useless listing?

To sum up, ego searches will remain a powerful way anyone measures the
relevancy of a search engine personally. However, they also remain a dangerous
way to assess the overall health and quality of a service. Ultimately, to
measure the quality of a service, you remaining wanting an entire battery of
different tests run, measuring relevancy and usability.

Related reading

mario-bros
analytics
Transaction data comparison
tracking clicks to offline