MobileSmartphone Search Engine Crawlers & Mobile URLs

Smartphone Search Engine Crawlers & Mobile URLs

With Google introducing the Googlebot for smartphones crawler and Bing embracing a “one URL per content item” strategy, life has become a lot easier in mobile SEO. However, you still need to watch out for these potential problems and common issues.

Example of different mobile phonesIn the mobile space, there has been much debate about mobile site structure: the costs and benefits of using multiple URLs or one URL to represent the mobile version of an existing page template.

My original stance was that the best option is to use one URL and multiple style sheets to control the rendering of the mobile content. As is always the case, that solution really didn’t suit everyone.

Developers, especially in the entertainment space, pointed out that the solution was only good for websites that focused on text; it wasn’t as good for media-rich, experiential websites.

Designers, especially those who were already getting into mobile app design, felt cramped by the conventions of traditional web layouts, and understood that mobile users related and responded to different design cues, that actually make information architecture and page organization easier on the smaller screen.

Despite the objections (whining?), I was an SEO at heart, and knew the value of an optimized URL, that had history and links already indexed in the search engines. So I stuck by my position, as much as I could, frequently suggesting server settings changes and DNS architecture to overcome most of the objections.

I still believe that “one URL” is the best-case scenario for mobile search results in many cases. Bing has recently stated publicly that they agree.

The biggest problem that my clients and I always faced in mobile SEO was that their desktop pages would almost always out-rank their mobile pages, especially in smartphone search. This was because the desktop pages were older, and had more links and more search engine credibility than the new mobile pages. Using one URL eliminated that problem.

Most of my clients however, had already built mobile content on a mobile subdomain (m.)or subdirectior (./m), so we had to set up user-agent detection and redirection on all desktop pages, and then work with canonical tags to ensure that the search engines understood that the two URLs represented essentially the same content.

In December, Google introduced the smartphone crawler (User agent string: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html), which made the life a mobile SEO a lot easier! The Smartphone crawler does what many of us have been instructing clients do with redirection or canonical tags for a couple years – but does it automatically!

The new crawler:

  • Associates desktop pages on a website with their mobile counterparts (halle-freaken-lujah!)
  • Records the mobile redirects that it finds from desktop to mobile pages
  • Seamlessly serves the target of the redirect (the mobile page) when a smartphone requested the desktop page from a smartphone search result.

It even does this without hitting your server for the redirect, because it (presumably) caches the two pages together, and does the disambiguation on its own.

While in some ways this seems like a fancy technical workaround by Google, because the problem could have also been handled algorithmically, it is still a great stride forward. This change is much better than some of the other paths Google could have followed, and allows mobile sites to have the best of both worlds (mobile and desktop designs) without forcing them into a one-URL-strategy.

While there still are some risks of mis-indexing or technical difficulty, this does potentially make mobile SEO a lot easier. Now, mobile pages will actually get the traffic they deserve, and servers will not be over-taxed with mobile redirection.

So now, if you are like most companies, and have built separate mobile pages, all you have to do is make sure that the smartphone bot is effectively triggering and crawling the redirects, (or if you are other companies, just check that the redirects are working properly in the first place!). If you don’t have redirects in place, and are relying on users to click over to the mobile content, or are using a JavaScript take-over to suggest people click to go to the other content, then you aren’t going to benefit from this new smartphone crawler.

Potential problems with the smartphone crawler enter the picture when you have done things in your robots.txt file to change how bots might index or approach your mobile content, or when you are using a system that generates temporary urls on the fly, rather than relying on a static mobile url to do the job. (This means look very closely at your setup if you are using a transcoding engine to generate your page – They commonly rely on temporary mobile urls which could cause problems for the smartphone crawler).

The other common issue that will hopefully be sorted out quickly by Google will be when desktop pages have more than one alternative mobile page; for instance one on the ‘m.’ for phones and another on a ‘t.’ for tablets. Google has yet to introduce a tablet crawler, so the best way to deal with this will really depend on testing.

Check that your desktop search results are redirecting appropriately across a number of phones, then see what happens when you do the same search, but click through from a tablet. It might work, and send you to the tablet-specific page, but if not, you might have to set up tablet user-agent detection and redirection from the mobile page that the smartphone bot has indexed to get visitors to the tablet pages. (If you need assistance with this, this mobile redirection script generator can help you create the exact code that you need to add to your site to make it happen).

The launch of the new Google smartphone crawler is interesting, and certainly bringing more needed attention to mobile SEO. The full impact of this new technology will vary depending on how your mobile content is set up, and what you have already done to optimize the user experience for your mobile site visitors.

Time will tell whether this will be a permanent mobile indexing strategy, or just another flash in the pan that Google only half-heartedly supports and eventually flames out. Hopefully it is a good sign of things to come in the world of mobile SEO and transparency from the Google mobile teams!

Resources

The 2023 B2B Superpowers Index

whitepaper | Analytics The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing

whitepaper | Analytics Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook

whitepaper | Digital Marketing The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

whitepaper | Digital Marketing Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y