About a month ago, Google Blogoscoped wrote of a link bomb that pushed UK prime minister Tony Blair's official biography to number one in a search for liar on Google. That is, if you use Google UK. Do the same exact search on Google.com, and the biography currently ranks fifth.
What's going on? It's not a case of having used the "Pages From The UK" option at Google UK to get UK-oriented results. This difference happens when using the default search "The Web" option at Google UK. In short, a search across the entire web at Google UK is not equal to a search across the entire web at Google's flagship Google.com site.
This isn't a Google-specific issue. The liar query underscores a big change that's snuck up on search users of several major search engines over the past year. Search engines have straying more and more into showing different results at the various country-specific versions they operate, even if country-specific results were not requested.
Sometimes the fact that a query may hit different data centers can be responsible for such differences. For example, being based in the UK, I generally hit Google's data centers that serve Europe, which can be slightly out of sync with those that may serve the East or West Coasts of the US. This is an age-old problem for search engines with multiple data centers. My AltaVista's Imperfect Mirrors article looks at the issue for that service back in 1998.
With Google and the Tony Blair link bomb, mirroring isnt' the case here. If it were, then my searches across the entire web would be the same at Google and Google UK.
Legal issues are another reason why you may see differences between country-specific versions. Search for nazi on Google France, and you won't find the American Nazi Party ranking first, as it does on Google.com. That's a result of Google having to comply with French censorship.
Head over to Google Canada, and the Jew Watch site that embroiled Google in controversy last year doesn't show up number two for a search on jew, despite the fact it is number two on Google.com. It also doesn't appear in Google France and Germany.
In Canada, Google tells me local laws have required that the site not be shown. That appears to be the case in France and Germany. Elsewhere, such as Google US, Google UK, and Google Australia, the site gains top rankings.
Aside from mirroring and censorship, the liar query on Google highlights a deliberate skewing of web results by Google to favor things its believes those in various countries would want to see. Our SEW Forums thread, Google Results Different Outside The US, covers more about this type of shift.
Here's an example. Search for toronto hotels on Google Canada. You'll see the results are ordered differently and some pages may not appear in the top results in both places.
The differences are generally small when you see changes between Google.com and Google country-specific results. One or two sites might be missing, and the rankings may be slightly different, but for the most part, things seem largely the same.
As for the user experience, you can understand the benefit for the UK user for the US National Football League site NOT to show up first for football, as it does for those using the largely US-aimed Google.com site. That's especially so given that we know searchers generally don't make use of little radio buttons or drop down boxes to get a country-specific experience.
In other words, all major search engines will let you get back results that are supposed to be aimed at a particular country, if you make use of special options. But since people don't do this -- and then get dissatisfied -- the trend seems to be to just make results country-specific to a lesser degree anyway.
While I understand the desire to automatically help users, I do find the country-skewing alarming. That's because I think most searchers have no idea that their country-specific results might be different than what others see.
They assume they were searching the web in the same way others were searching the web, without some type of country flavoring. In fact, I've corresponded with some search experts and marketers who've made this mistake. It's a trap I often have to remind myself not to fall in, either. If skewing is happening, it needs to be made clearer to searchers.
Looking at Google UK, I could find nothing in its help pages that suggest any type of country skewing is happening there. How about elsewhere?
Yahoo & Skew
Over at Yahoo, it says that things like link data, language information and other factors are used to make results a little more country-specific at a particular country version, even if you don't explicitly ask for this to happen.
For example, a search for cars on Yahoo UK brings up slightly different web search results than at Yahoo.com. Results for "liar" are also slightly different -- though Tony Blair doesn't come up in the top results on either version.
Paid inclusion can also play a role, but only a minor one, the company says. Those in the paid inclusion program can choose to target certain URLs toward people in particular country, so that's another factor. However, Yahoo says it should have only a small impact, as paid inclusion URLs make up only a tiny portion of its index.
What's the disclosure of this to searchers? At Yahoo, there isn't any. The Advanced Web Search page provides the ability to narrow results down to a particular country, which implies that if you don't do this, you'll search the entire web. And that's exactly what Yahoo's help page about advanced search says should happen:
Country: Yahoo normally searches through websites from all over the world. You can choose to see sites just from a particular country.
In reality, it may search through web sites worldwide, but exactly which pages and how they are ranked can vary depending on the edition you use.
Going forward, the skewing isn't going to change, as Yahoo finds good reasons to help users in doing this. However, Yahoo said it thought more information could be provided to inform users that country skewing happens via its help pages. So that might come.
MSN & Skew
MSN does have an entire section in its help area discussing country skewing: Locate content by language or country/region. It says right up front that the country you are searching from will be used as data for generating search results:
We fine-tune our search to provide you results that are specific to each locale. When you first start using MSN Search, we automatically detect your location by looking up your computer's Internet protocol (IP) address. We then search for results in the MSN Search site that is designated for your location.
The solution if you don't want that skewing? MSN says to use the site it offers for other regions or countries. OK, fair enough. But if there's going to be an "Only from the United Kingdom" checkbox at the bottom of MSN Search UK, it still seems like that's not going to clue some people in that the results are already being skewed. Somehow, someway, this should be raised to more attention.
By the way, MSN says another option to avoid skewing is to change your location via the Settings option. However, I found that still doesn't change the results -- and you can only do it on the US site, from what I can see. In other word, Settings doesn't give me an location options for MSN UK. If I go to MSN US, then I see my location showing as London. I can change that to Newport Beach, California, but going then back to the MSN UK site still brings up UK skewed results.
Searchers should consider the Search Builder option on the MSN home page, however. With that, you can build up pages deemed relevant to multiple countries of your choosing, including the United States. Languages can also be targeted. Other search engines offer similar functionality on their advanced search pages.
Ensure The Site Is The Same For Everyone
I doubt skewing will go away. I do think that if it's here to stay, the search engines should at least assure that it doesn't happen if you go to a country-specific edition. In other words, everyone who goes to Google UK should see the same thing, regardless of whether they are in the UK or not. The same for those going to Google US.
This is in terms of editorial results, of course. In terms of ads, I think the search engines should provide options. If you want to see ads targeted at those in particular countries, when at a particular search engine, you should be able to say so. As an American who lives outside the US, I still have an interest in US-targeted ads. I'm hardly the only expat of any country like this.
What's a webmaster to do? Understand that skewing won't go away. It's always been a case that if you wanted to target different countries, you should ideally have a web site for each country. That's even more so, going forward. If you have only one site and aim toward a global audience, chances are you're going to find that country-specific skewing may ultimately work against you.
Want to do well for a particular country? Traditionally, the ingredients you need are these:
- Use a country-specific domain name
- Write in the language of the country
- Gain links from sites that are clearly seen as belonging to part of that country
- Considering hosting your site physically within the country
I also asked each search engine for any specific advice, in particular how to avoid possible duplicate content problems if they want to have an English language site that targets different English-speaking countries.
Google said that it is rare to hear of a duplicate content issue. In other words, say you have a site with a page written in American English, but you want to use the same page without changing to UK English on your UK web site. Bad move, in terms of making the UK audience happy! But Google said you'll almost certainly be fine.
The key exception is if you have virtually no content. For example, say you have a one page web site written in English. You then decide to register domains for the UK, Australia, Canada and New Zealand in hopes that will help you bring in an audience from those places. Google says it would be best to redirect those domains to your main .com web site rather than running them all independently, given that you have so little content.
Google's also big on links, point three in the list above. Build links from sites in the countries you wish to target, pointed at your own country-specific web site, and that should help.
MSN said it's best to establish a country-specific domain for the various markets you wish to target. So if you wanted to target the US and the UK, you would have something like company.com and company.co.uk. Then you should permanently redirect people to your main site. So those in the UK -- you might still promote the company.co.uk domain in that country and in links, but they'd be redirected to your .com site.
Personally, unless I had a very small site with little content (as with the Google example above), I'd do the alternative. I'd run two separate sites and trust that the minor differences between US and UK English would help avoid any penalties, along with the fact that I might have some content unique for my UK visitors, such as store location or certain products only offered there.
Yahoo said the above bulleted tips are all useful for them. A country-specific domain name isn't required but will help, it said. It also referred readers back to an old SearchDay article on how Inktomi did country-targeting: Inktomi Improves International Search. So check that out.
Looking for more help? See our Multilingual Search Markets & Non-US Engines section of the Search Engine Watch Forums for discussions on this topics. Below are some interesting threads from that forum area, along with related reading.
- ca.yahoo.com vs yahoo.com (forum discussion)
- Extending your Search Marketing Campaign to Europe (forum discussion)
- Can Google Really Deliver Country Specific Searching?
- UK Versus US & Other English Versions (forum discussion)
- Cultural Differences by Country (forum discussion)
- Escaping MSN Search's Country Redirection
- Got To Censor Search Listings? Why Not Disclose?