Testing the International Googlebot, Simplified

Last month we discussed how, in an attempt to keep up with the rising trend of sites dynamically changing content depending on the location or language preference of their customers, Google has started crawling from non-U.S. IP addresses. Google now requests URLs with Accept-Language headers to discover content that is locale-adaptive.

Because it’s important for SEOs to stay on top of the latest changes Google is making – and due to the overwhelming response received – the previously-featured testing tool has been improved to make it even easier to discover how sites are configured for the international locale-aware crawling of Googlebot. It’s now not only possible to check if your content changes based on the Accept-Language headers, but also from several non-US IP locations.

Step 1 – Specify the URL to Test

Begin by simply specifying the URL you would like to test. Here, we choose Google Cloud.

locale-adaptive-crawling-test-specify-urlStep 2 – Check Languages

Next, specify the languages you’d like to test against. This will send the proper Accept-Language HTTP request header accordingly. In the example, below we’d like to check if Google Cloud is dynamically changing content for English versus Japanese speakers, and if Googlebot sees the change, as well.

locale-adaptive-crawling-test-check-languages

Step 3 – Specify Googlebot & Country of Choice

Finally, before clicking Run, choose Google from the drop-down menu and the country of your choice. Here, we’re checking to see how Google Cloud is dynamically changing content for users in Japan that speak English versus Japanese.

locale-adaptive-crawling-test-googlebot-and-country

Evaluating Results

Now, we can evaluate the results. In the upper right-hand corner, you can see that the results can be exported to Excel. The column headers seen below are defined as:

  1. Accept-Language. This was the Accept-Language headers, chosen in Step 2.
  2. HTML does not equal request 1. Did the HTML change from the initial request? This is meant to be an indicator that the site is sending different HTML source code. A simple character difference – a dot instead of a commaa – will return a “Yes.”
  3. Detected Language. This is the language detected on the page, according to Google’s a language detection API.
  4. Status. This is the returned status code based on the HTTP Header response.
  5. Content-Language. This returns what is specified as the intended language in the Content-Language HTTP header; remember, using specifying in the < meta> data is not recommended.
  6. Vary: Accept-Language. “Yes” or “No” answers whether the server sends the Vary: Accept-Language response header. If a URL is locale-adaptive, it’s a good idea to add a Vary: Accept-Language response header. This should help Googlebot understand that different content is available for different language requests.
  7. Redirect. This populates the new location if the page redirects.
  8. Page Title. This pulls the < title> element’s content from the source code.
  9. Meta Description. This pulls the meta description.
  10. H1. This pulls h1.

locale-adaptive-crawling-test-evaluating-results

Conclusion

The Internet has connected the world in fascinating ways. As this continues on an exponential scale, SEOs who become experts in the nuance of international SEO will certainly reap the benefits.

Remember that if you’re checking whether the URL is changing content based on the IP, choose “none” as the language. Choosing languages can send mixed signals. The page can be locale-adaptive, but you won’t know if it’s based on the language or the IP. However, after testing signals independently, running the test with languages and proxies will enable you to see which signal trumps the other.

In this example, cloud.google.com is only using the Accept-Language, not the IP. Facebook is an excellent example of a site that prioritizes Accept-Language first and then falls back to IP if there’s no language preference specified by the customer.

Related reading

dna34_hotels_venice_beach
local map pack
thinkstockphotos-507274998
local pin
Simple Share Buttons