5 Reasons the Google Search Bias Study is, um, Biased

Increasingly, the quality of Google’s search results and business practices are being called into question. A new study co-authored by Benjamin Edelman and Benjamin Lockwood attempted to measure bias in the organic search results of Google and to a lesser extent Yahoo, Bing, AOL, and Ask.

Claims of bias are nothing new to Google. In the European Union, Google is facing increasing antitrust scrutiny from regulators over its search practices — with the possibility that Google will have to give the EU access to its algorithm.

In a previous Edelman report, he accused Google of hard-coding results to favor itself over vertical competitors. The assistant professor at Harvard Business School (who, it should be noted, is involved on a lawsuit against Google, is a member of the Alliance Against Bait and Click, and also has ties to Microsoft) has also authored several other interesting pieces on Google, Facebook, and more over the years.

So does this study of the organic results, based on August 2010 data, finally reveal once and for all without a doubt that Google’s results are clearly biased? No. Here are five reasons why.

1. Google: More than Just a Search Engine

Google is much more than just a search engine. YouTube, Maps, News, Apps, Chrome, Product Search — Google offers several products, many of which the other members of the top five search engines simply don’t have (e.g., Ask couldn’t get preferential treatment for Ask Mail and AOL couldn’t favor AOL Maps because no such products exist).

Google is ultimately a business. However, Edelman’s study says Google should be held to a higher standard because it “claims that its algorithmic results are ‘algorithmically-generated’, ‘objective’, and ‘never manipulated’.”

  • Algorithm: Google says it uses more than 200 signals to order websites and updates these algorithms weekly. Plus there is personalized search based on a user’s search history. Part of the rationale behind Google Instant was that Google wants to anticipate what users want, based on data they’ve already collected about you from search history and by using Google’s products, combined with location information and the huge amounts of data they’ve collected on everyone.
  • Objective/Never Manipulated: Edelman links to this page, which reads: “Advertising on Google is always clearly identified as a ‘Sponsored Link,’ so it does not compromise the integrity of our search results. We never manipulate rankings to put our partners higher in our search results and no one can buy better PageRank. Our users trust our objectivity and no short-term gain could ever justify breaching that trust.”

OK, key point here: Google is not a partner. Google is Google. Google isn’t claiming that they don’t manipulate their rankings to help themselves.

Google probably legitimately thinks Google Maps is better for its users than other websites. According to this study, Bing must think so, too. But not Yahoo…

2. Google vs. Yahoo: The Bias of Bias


By looking at the figures in this table, Yahoo is really the bigger offender here and has a higher probability of linking to its own service. Which really doesn’t matter anymore, because now Bing serves Yahoo’s organic results.

Calling a survey “Measuring Bias in ‘Organic’ Web Search” and focusing mainly on Google while glossing over Yahoo when the results clearly show it is the bigger offender of “organic bias” seems a bit sketchy, as if they came to a conclusion and then bent the statistics to back it up.

Edelman explained that, in addition to the ‘algorithmically-generated’, ‘objective’, and ‘never manipulated’ reasoning, they focus on Google because of their dominant market share.

What’s the solution? For Google to reveal every detail of how the algorithm works?

All that would happen is the competitors would copy the secret sauce, resulting in even worse gaming by webmasters, according to James Grimmelmann, an associate professor at New York Law School, who wrote an interesting paper about search neutrality (PDF here).

3. Bias By the Numbers

Google was ranked No. 1 in its own search results for 11 of the 32 searches (34 percent); No. 2 six times (18 percent); No. 3 four times (12 percent); and not ranked at all 13 times (37 percent) — math note: Google appeared twice on the

search in first for Google Videos and third for YouTube.

Yahoo was ranked No. 1 on its own search results for 11 of the 32 searches (34 percent); No. 2 eight times (25 percent); No. 3 twice (6 percent); and didn’t rank at all 17 times (53 percent). Yahoo has the distinct honor of owning the top three results on a search for [calendar] and pushed its own services twice on four of those searches [finance], [health], [photos], and [stocks].

Bing was a complete no-show here, not ranking any Bing/Microsoft properties in its top three organic results.

Looking at these numbers, clearly this data is irrelevant and proves nothing. It’s a false argument to suggest that if Bing doesn’t link to a Google product, therefore Google’s results need oversight.

Why? Every algorithm is different.

So out of an available 90 spots of most valuable SERP real estate, both Google and Yahoo occupied 21 of them, or 23 percent. Bing, the search engine saint, wastes none of that space on sending traffic to its own properties.

Oh, did I say saint? How about this:


Hey, look! A sponsored site above the organics? Wonder how much Microsoft paid for that ad. Perhaps Bing/Microsoft didn’t place itself into its own organic results because they were using this section to promote their own products?

Maybe somebody will do a study on that. I won’t hold my breath.

4. Click Fraud

It’s no secret that top search results get the most clicks. So Edelman’s study also looked at click-through data and determined that “Across all search engines and search terms, the first result received, on average, 72% of users’ clicks, while the second and third results received 13% and 8% of clicks, respectively.”

Interestingly, in Google’s results, the study found that for [email], Gmail, the first result, receives 29 percent of users’ clicks, while Yahoo mail, the second result, receives 54 percent.

One possible explanation for this: Google links to Gmail in the top nav bar. Surely that has some impact on why people using Google to search for mail wouldn’t need to click on Gmail in the SERP.

Also, organic results and clicks don’t take into account direct traffic or links from other websites, and thus wouldn’t show up as a factor here in determining popularity.

As mentioned, the study is based on 32 searches. Granted, the focus here was to “catch” Google listing its services first and getting all the clicks even though another site may be more popular or helpful.

But in the scheme of things, 32 searches is a tiny number for ultimate proof — just as you can’t prove a superior search engine based on 20 searches.

How about this number: 10.3 billion. That’s how many searches were conducted on Google in August, when this data was gathered.

Search isn’t an exact science after all.

5. Search is Subjective

Their big claim is that they have discovered “Google’s algorithmic search results link to Google’s own services more than three times as often as other search engines link to Google’s services.” In an e-mail, Edelman explained that the purpose of the study was to examine “whether Google links to Google Maps more than others link to Google Maps,” for example.

To get an idea of whether there are some shenanigans at work, let’s take a quick trip to Yahoo Site Explorer. My top three results in Google for [maps] look like this:

  1. Google Maps (inlinks: 2.5 million)
  2. Yahoo Maps (inlinks 5.8 million)
  3. Bing (inlinks: 13,000)

My personal [maps] results differ than what Edelman found. However, it’s also five months later, and I have an entirely different search history/profile with Google, so without a long-term analysis (months or even better years of data), search results are a photograph of a moment in time.

Bing isn’t obligated to link to Google services just as Google isn’t obligated to link to Bing, but in this simple example, they are!

Users aren’t forced to pick SERP result No. 1 by default. If people don’t like the results, let them switch over to Bing, DuckDuckGo, or Blekko.

People using Google recognize that, for the most part, Google delivers useful products. Perfect? Hardly.

But to expect Google to be punished because they built a better product isn’t a solution, and neither is government intervention. We don’t need a “War on Search Results” to add to the list of failed government wars on concepts. Let’s not go the route of the Chinese search model.

Google may be popular target of government agencies, regulators, and even the SEO/tech community at the moment. But whatever we think of Google, the company is still raking in billions every quarter and users haven’t started abandoning their search product in droves (they just gained market share last month, again).

This whole study is a fallacy wrapped in manipulated statistics by someone with a vested interest in Google’s failure.

Edelman no doubt will be cheering on the European Commission, based on the closing sentiment of the study. Measuring bias, indeed.

Related reading

Simple Share Buttons