MSN’s Hand Crafted Results (Fake? – Shame On Me!)

"MSN
Hiring People to Hand Code SERPS
" at SEO Blackhat is a nice catch from the
MSN Search jobs
page
talking about needing people to help hand-craft results. Philipp Lenssen at
Google Blogoscoped reacts with "Oh my." I react with "Hallelujah."

Note: As Threadwatch
spots
in comments, this page looks like a joke that MSN is hosting. Shame on
me for not reading more closely — type 150 words per minute! The page IS on the
real MSN Search domain, but it’s not linked from the real
jobs area [OK,
Pip at Google Blogoscoped
found
it connected from the jobs page]. Anyway, I’ll drop a note and get
confirmation. And the points below — still valid :)

Let’s look at the job post first:

When all else fails, and the ranking algorithms do not pass the confidence
threshold, we fall back to delivering handcrafted results. Working on a team of
approximately 132 other handcrafters in 26 worldwide markets, you will receive a
user query, use all the available search engines to quickly scour the web for
results, pick the top 10 results for this query, and send it on to the user.
Successful handcrafters can typically find top 10 results for a real-time user?s
query in less than 3.8 seconds. This is an opportunity to truly connect with
customers, because the queries that get routed to you are precisely the ones
that the engine cannot answer well. We will have adequate staffing to allow
generous coffee and bathroom breaks. If you are an expert at using at least 3
different search engines, well versed with American English/colloquial usage,
and can type at > 149 words/minute as measured by the Simia-Lico method ? come
join us and delight users real-time!

I agree. Search engine algorithms are not perfect. I’m tired of seeing bad
listings make it into the top results that any human reviewer would nix. The
Google mantra has always been that they prefer to tailor their computer
algorithms to figure out how a human would see and rate things and then get the
algorithm to do the right thing. We’ve had that mantra for years. And yes,
generally the algorithms do the right things. Still stuff gets through. So kill
off the bad stuff with a human and sure, insert a good quality page you know you
are missing.

As a reminder, MSN
used to have
human editors, as I’ve written before. That was actually one reason why years
ago, they compared pretty well when we would do relevancy tests on popular
queries. They had a very sophisticated editing suite that allowed a team of
editors to constantly review — AND FIX — bad results.

Now I can buy into the "Oh My" idea if MSN is returning to hand crafted
results because their automated technology is so bad they’ve got to fall back on
humans. No, that’s not good. But if it’s to complement and better tune what the
automation does? Bring it on. If you want more on the how and why this can help,
see my past post,
More On Query Refinement, The Human Scale Problem & Creating The Search Dialog
.

I also have the "Oh My" reaction if hand handcrafting involves payment. This
year, I’ve had one serious allegation that MSN has rigged one set of its results
to favor a top advertiser. I just had another serious allegation like that
levied against Yahoo. In the MSN case, the difficulty in pursuing the allegation
is deciding whether they are true or an attempt to knock out a competitor that
might be ranking well. In the Yahoo case, I’m awaiting that tipster to send me
more information beyond the quick eye opening stuff I was shown at our recent
London conference.

Yahoo, of course, does hand manipulate already, to my belief (I’m not saying
for payment — only that for whatever reason, they seem to hand craft some
results). I
wrote
about this in 2004 but never got an answer about it from Yahoo, nor
did I get an answer when I followed up at least one other time. It also
came up
on our forums last year and at
here at
Search Engine Roundtable.

Google has long denied "hand jobs," as wizened search marketers call them.
Setting aside censorship cases, I believe that. I’ve never seen any solid
evidence of results being hand selected by Google (and the quality raters we’re
written about
before
have not been shown to be manipulating results).

In fact, Google used to trumpet that it had no hand manipulation. That was
true in crafting results, but it wasn’t true in terms of removing them. As I
wrote
in 2004:

Of course, Google does indeed
intervene manually in search results. It removes material because it may be
deemed illegal, as was the case in the infamous
chester
guide
search. The company also removes material in response to DMCA
complaints and also because for spamming reasons, as this
article
explains further.

Such interventions make some
marketers confused (or even livid) when they read Google’s oft-repeated claims
of no hand manipulation of search results. To them, such removals as I’ve
described above are hand manipulation. You can get a flavor of such confusion
in this recent WebmasterWorld
forum thread.

These interventions are not
specifically rank related. When they happen, Google doesn’t try to reorder the
ranking of how a page appears. Instead, it simply pulls the page from the
index entirely. And if you aren’t in the index, you naturally no longer rank
number one. But to save confusion, it might be better for Google to be clearer
in saying that they don’t chose by hand which sites rank well.

By the way, I asked Google
previously about the reference in a Wired
article
about wanting to "attach" better sites to queries to ensure it had good
information available. I remember being disturbed by this, just as some in the
aforementioned thread were, as it indeed suggested that Google was doing
hand-ranking in some cases.

I was told by Google that this
was a misinterpretation on the part of Wired. The Google engineer apparently
meant that the Google search algorithm would be tweaked to produce better
results, not that the results would be reordered by hand.

Overall, I’m fine with hand-crafting, hand manipulation, hand jobs or
whatever you want to call it as long as:

  • It improves search quality
  • It’s not done to favor an advertiser by rigging the editorial results

Related reading

Google I O 2016 on Google
A graphic showing the old Google logo superimposed over a faint search results page.
Shift London
471de69559512ff4f56b896ccedc6389