MSN's Hand Crafted Results (Fake? - Shame On Me!)

"MSN Hiring People to Hand Code SERPS" at SEO Blackhat is a nice catch from the MSN Search jobs page talking about needing people to help hand-craft results. Philipp Lenssen at Google Blogoscoped reacts with "Oh my." I react with "Hallelujah."

Note: As Threadwatch spots in comments, this page looks like a joke that MSN is hosting. Shame on me for not reading more closely -- type 150 words per minute! The page IS on the real MSN Search domain, but it's not linked from the real jobs area [OK, Pip at Google Blogoscoped found it connected from the jobs page]. Anyway, I'll drop a note and get confirmation. And the points below -- still valid :)

Let's look at the job post first:

When all else fails, and the ranking algorithms do not pass the confidence threshold, we fall back to delivering handcrafted results. Working on a team of approximately 132 other handcrafters in 26 worldwide markets, you will receive a user query, use all the available search engines to quickly scour the web for results, pick the top 10 results for this query, and send it on to the user. Successful handcrafters can typically find top 10 results for a real-time user?s query in less than 3.8 seconds. This is an opportunity to truly connect with customers, because the queries that get routed to you are precisely the ones that the engine cannot answer well. We will have adequate staffing to allow generous coffee and bathroom breaks. If you are an expert at using at least 3 different search engines, well versed with American English/colloquial usage, and can type at > 149 words/minute as measured by the Simia-Lico method ? come join us and delight users real-time!

I agree. Search engine algorithms are not perfect. I'm tired of seeing bad listings make it into the top results that any human reviewer would nix. The Google mantra has always been that they prefer to tailor their computer algorithms to figure out how a human would see and rate things and then get the algorithm to do the right thing. We've had that mantra for years. And yes, generally the algorithms do the right things. Still stuff gets through. So kill off the bad stuff with a human and sure, insert a good quality page you know you are missing.

As a reminder, MSN used to have human editors, as I've written before. That was actually one reason why years ago, they compared pretty well when we would do relevancy tests on popular queries. They had a very sophisticated editing suite that allowed a team of editors to constantly review -- AND FIX -- bad results.

Now I can buy into the "Oh My" idea if MSN is returning to hand crafted results because their automated technology is so bad they've got to fall back on humans. No, that's not good. But if it's to complement and better tune what the automation does? Bring it on. If you want more on the how and why this can help, see my past post, More On Query Refinement, The Human Scale Problem & Creating The Search Dialog.

I also have the "Oh My" reaction if hand handcrafting involves payment. This year, I've had one serious allegation that MSN has rigged one set of its results to favor a top advertiser. I just had another serious allegation like that levied against Yahoo. In the MSN case, the difficulty in pursuing the allegation is deciding whether they are true or an attempt to knock out a competitor that might be ranking well. In the Yahoo case, I'm awaiting that tipster to send me more information beyond the quick eye opening stuff I was shown at our recent London conference.

Yahoo, of course, does hand manipulate already, to my belief (I'm not saying for payment -- only that for whatever reason, they seem to hand craft some results). I wrote about this in 2004 but never got an answer about it from Yahoo, nor did I get an answer when I followed up at least one other time. It also came up on our forums last year and at here at Search Engine Roundtable.

Google has long denied "hand jobs," as wizened search marketers call them. Setting aside censorship cases, I believe that. I've never seen any solid evidence of results being hand selected by Google (and the quality raters we're written about before have not been shown to be manipulating results).

In fact, Google used to trumpet that it had no hand manipulation. That was true in crafting results, but it wasn't true in terms of removing them. As I wrote in 2004:

Of course, Google does indeed intervene manually in search results. It removes material because it may be deemed illegal, as was the case in the infamous chester guide search. The company also removes material in response to DMCA complaints and also because for spamming reasons, as this article explains further.

Such interventions make some marketers confused (or even livid) when they read Google's oft-repeated claims of no hand manipulation of search results. To them, such removals as I've described above are hand manipulation. You can get a flavor of such confusion in this recent WebmasterWorld forum thread.

These interventions are not specifically rank related. When they happen, Google doesn't try to reorder the ranking of how a page appears. Instead, it simply pulls the page from the index entirely. And if you aren't in the index, you naturally no longer rank number one. But to save confusion, it might be better for Google to be clearer in saying that they don't chose by hand which sites rank well.

By the way, I asked Google previously about the reference in a Wired article about wanting to "attach" better sites to queries to ensure it had good information available. I remember being disturbed by this, just as some in the aforementioned thread were, as it indeed suggested that Google was doing hand-ranking in some cases.

I was told by Google that this was a misinterpretation on the part of Wired. The Google engineer apparently meant that the Google search algorithm would be tweaked to produce better results, not that the results would be reordered by hand.

Overall, I'm fine with hand-crafting, hand manipulation, hand jobs or whatever you want to call it as long as:

  • It improves search quality
  • It's not done to favor an advertiser by rigging the editorial results