Throughout 2013 and into 2014, Google has been making various improvements to the way it handles website content on mobile devices. That continues this week with the announcement of a new user-agent for crawling smartphone content.
Google says it’s retiring “Googlebot-Mobile,” for smartphones, which has been used to refer to various mobile-specific crawlers that index content for both feature phones and smartphones. It will be going away within three to four weeks. After that, the user-agent for smartphones will simply be identified as “Googlebot,” but will list “mobile” elsewhere in the user-agent string.
Google shows the difference between the new and old user-agents:
<
Google says it has seen cases where webmasters have inadvertently blocked smartphone crawling while trying to only block feature phone crawling.
“This ambiguity made it impossible for Google to index smartphone content of some sites, or for Google to recognize that these sites are smartphone-optimized,” explains Google smartphone search engineer Zhijian He.
The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag and HTTP header directives for Googlebot as opposed to Googlebot-mobile.
The update, He says, affects less than 0.001% of URLs, based on Google’s internal analysis.
Webmasters can, of course, test their sites with the Fetch as Google feature in Webmaster Tools.
Last month, Google expanded the Crawl Errors feature in Webmaster Tools to help webmasters identify pages on their sites that show smartphone crawl errors. Earlier in 2013, Google made several ranking changes for sites not configured for smartphone users.
Google also started indexing in-app content, so smartphone users searching can access deep links in mobile apps from the search results page.
Image via Google