Mobile SEO: Managing Googlebot & Your Mobile Sites

                                                                                                           

 Google announced that it now has a smartphone Googlebot-Mobile. Historically, Googlebot-Mobile has concerned itself only with mobile sites designed for feature phones, so this is a pretty significant development. Today's column is going to discuss how to think about Googlebot, Googlebot-Mobile and your mobile web site.
As noted in this recent mobile SEO podcast with Cindy Krum one of the first decisions you need to make is what types of devices you’re targeting. If you don’t have many feature phone users who are trying to access your site, you may not need to support them, and you can focus your mobile phone site on smartphone devices.
This impacts how you set up your site. Mobile sites that don’t support feature phones should use a same URL strategy - i.e. one where the mobile content renders on the exact same URL as the desktop content, and smartphone users are served the mobile version of the site using user agent detection (we call this the "Same URL" strategy). The reason you would prefer this is that your mobile site will then inherit all the SEO benefits of your desktop site (i.e. the link profile and other measures of content value and importance).
If you do support feature phones the problem becomes more complex. The technical challenges in supporting a wide array of form factors / screen sizes might make it easier from a technology standpoint to build your mobile site using a m.yourdomain.com approach. You lose the SEO goodness of the desktop site, but you can still send users to it using user agent detection.


                                                  

Googlebot-Mobile Detection

Once you have decided on an approach, you also need to set up your user agent detection. This is the process by which you recognize incoming user agents. When you see an incoming user agent that is mobile device specific, send them to the mobile version of your site. Make sure that your user agent detection includes Googlebot-Mobile. Google just introduced a new version specific to mobile sites designed for smartphones. The current user agent strings used by Googlebot-Mobile are:

I asked Google for clarification on how they recommend the user agent detection for their crawler should be implemented. A Google spokesperson told me:

"There are two parts to this: The user agents of Googlebot-Mobile contain device names that represent certain classes of mobiles. We have two for feature phones and one for smartphone. They're all listed in the blog post you referenced.

Conceptually, our recommendation is this: Take the user agent Googlebot-Mobile has specified, remove (or ignore) the Googlebot identifying part. This leaves a device name representing a class of devices and websites should serve the best content they have for that class of device. For example, when you do this using the user agent of Googlebot-Mobile for smartphones, this kind of check will reveal an iPhone user agent and you should serve the content you have optimized for iPhones.

The corollary consideration is when the site does not have optimized content. In this case a website should serve what they would serve anyone by default. Usually that turns out to be the desktop content.

           

One last note: our Googlebot crawlers can all be verified/authenticated, including the three Googlebot-Mobile crawlers. This is described in our Help Center".

Here’s a summary of the process for detecting Googlebot-Mobile:

The above graphic is only for detecting Googlebot-Mobile, and your user agent detection will need more to it to pick up other user agents used by users. You don't need to detect the feature phone specific user agents if you don't intend to support feature phones. If you support both smartphones and feature phones with the same mobile site, you should redirect those to the same place.
Note the recommendation that you don't need to look for Googlebot-Mobile in the user agent string. The design of the Googlebot-Mobile user agents is such that you don't need to do that. For example, if your user agent detection routine is already looking for "iPhone" you are already picking up smartphone Googlebot-Mobile.
Now you have a few paths you can go down. These are captured in the following table:

Mobile Site Design Focus	Implementation Approach	User Agent Processing
1. Smartphones & Feature Phones	m.yourdomain.com	Send all 3 Googlebot-Mobile user agents to the mobile pages.
2. Smartphones & Feature Phones	Same URL	Serve both Smartphone and feature phone users the mobile pages, and don't let Google transcode the content for feature phones.
3. Smartphones Only	Same URL	Don't serve feature phone user agents (including the Googlebot feature phone user agents) the mobile pages. Let Google transcode the content for feature phones.

The key difference between options 2 and 3 is whether or not you think the smartphone version of the site is a better experience for feature phone users than Google's transcoded pages. If you are not familiar with transcoding, this is what Google does with feature phones that are trying to access web pages designed for desktop browsers – it transcodes that content into something it hopes will offer the feature phone user a better experience.

One Last Consideration

When you first put up your mobile site, you may want to tag the pages to let Googlebot (the normal desktop version) know about it. The tag looks something like this:

< link rel="alternate" media="handheld" href="alternate_page.htm" />

This tag is documented on this Google page. The page discusses transcoding, and the article there isn’t entirely clear on whether this tag is needed when you implement proper user agent detection. A lot of publishers have both desktop and mobile versions of sites that don't use the tag, so we asked Google two questions:

1. "Is there a reason to use this instead of, or together with, UA detection? Or is just an alternative way to redirect Googlebot-Mobile?"
2. "Is this targeted at feature phones or smartphones, or both?"

Here is what they said:

"This tag is relevant only for feature phone optimized websites. We recommend using it even when you do user agent detection and HTTP redirects, because thetag can be seen by normal Googlebot (and other search engines' crawlers) and can be used to improve the search experience even before our Googlebot-Mobile for feature phones reaches your site".

Summary

Implementing a mobile site is becoming increasingly important for many publishers, as smartphones are becoming increasingly pervasive. Today, it may be that the biggest benefit of this is in offering a superior user experience to mobile device users. While Google maintains a separate index for feature phones, the index for smartphones is the same as it is for the desktop
However, this is already beginning to change. Google has already indicated that they have a "Skip Redirect" function, where users on a smartphone device will see the mobile version of the URL in the results instead of the desktop version of the URL (this applied to m. implementations). Here is how they worded it:

"When we discover a URL in our search results that redirects smartphone users to another URL serving smartphone-optimized content, we change the link target shown in the search results to point directly to the final destination URL. This removes the extra latency the redirect introduces leading to a saving of 0.5-1 seconds on average when visiting landing page for such search results".

You can expect to see many more changes in the future, both to the presentation of the results, and the ordering of them as well.

---

You are invited to participate in a ClickZ-Google Analytics industry survey for trends in mobile marketing and apps. You'll also be entered to win a free iPad or 1 of 2 free passes to SES Conference & Expo.