Google announced that it’s introducing new locale-aware crawl configurations for Googlebot for pages it detects may adapt their content based on the request’s language and perceived location.
“Locale-adaptive pages change their content to reflect the user’s language or perceived geographic location,” Google says in a blog post. Since, by default, Googlebot requests pages without setting an Accept-Language HTTP request header and uses IP addresses that appear to be located in the USA, not all content variants of locale-adaptive pages may be indexed completely.”
The new configurations are geo-distributed crawling and language-dependent crawling. The former sees Googlebot starting to use IP addresses that appear to come from outside of the U.S. as well as the current IP addresses that appear to be from the U.S. that Googlebot already uses. The latter is where Googlebot crawls with an Accept-Language HTTP header in the request.
The new configurations are enabled automatically for pages Google detects to be locale-adaptive, and the company warns you may notice changes in how it crawls and shows your site in search results even if you haven’t changed your CMS or server settings.
Google supports and recommends using separate locale URLconfigurations and annotating them with rel=”alternate hreflang annotations. It considers using separate URLs the best way for users to interact and share your content, while maximizing indexing and better ranking your content.
More on locale-aware crawling by Googlebot here.
Image via Google