Google crawler is presented with desktop site when requesting mobile

[aredondo]

I am using mobile-fu in a web site that shows two different user interfaces to desktop and mobile users. I've noticed that for some reason, Google only recognizes the desktop version of the site—mobile-friendly sites are marked as such in the search results, and this one isn't.

Using Google Webmaster Tools, I've been able to confirm that indeed Google only receives the desktop version of the site, even when it places the request as a mobile device. Digging into this issue, the problem seems to be because of the Accept header that the crawler sends.

I've captured the request sent by the crawler, and it's like this:

GET / HTTP/1.1
Host: www.host.com
Accept: */*
Cache-Control: no-cache
Connection: Keep-alive
From: googlebot(at)googlebot.com
Accept-Encoding: gzip,deflate
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Now, the culprit seems to be the set_mobile_format function in Mobile Fu. In order to set the view as mobile, four conditions must be met—being the first one that the request is for html content, which is not the case for this request made by the Google crawler, as the content of the Accept header is */*.

Before patching this on my side, I would like to ask about two of these four conditions just to make sure I don't run into a bigger problem:

• Why is the mobile view only set for requests for html content? Is it just to avoid issues with, for example, API-type requests such as json or xml, or is there another reason?
• Why is the mobile view not set for xhr requests? Since the first condition ensures that the request is for html content, it'd make sense to me that returned content would be different for desktop browsers and mobile devices.

Would you be interested in a patch for this? Since these conditions seem to be so deliberated, I don't know if you'd consider this issue a defect.

Thank you in advance for any light you can shed about this.

[benlangfeld]

On 26 Dec 2014, at 04:04, Alvaro R [email protected] wrote:

I am using mobile-fu in a web site that shows two different user interfaces to desktop and mobile users. I've noticed that for some reason, Google only recognizes the desktop version of the site—mobile-friendly sites are marked as such in the search results, and this one isn't.

Using Google Webmaster Tools, I've been able to confirm that indeed Google only receives the desktop version of the site, even when it places the request as a mobile device. Digging into this issue, the problem seems to be because of the Accept header that the crawler sends.

Thanks for making the effort to debug this.
I've captured the request sent by the crawler, and it's like this:

GET / HTTP/1.1
Host: www.host.com
Accept: /
Cache-Control: no-cache
Connection: Keep-alive
From: googlebot(at)googlebot.com
Accept-Encoding: gzip,deflate
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Now, the culprit seems to be the set_mobile_format function in Mobile Fu. In order to set the view as mobile, four conditions must be met—being the first one that the request is for html content, which is not the case for this request made by the Google crawler, as the content of the Accept header is /.

Before patching this on my side, I would like to ask about two of these four conditions just to make sure I don't run into a bigger problem:

Why is the mobile view only set for requests for html content? Is it just to avoid issues with, for example, API-type requests such as json or xml, or is there another reason?
You are correct here. This case should be treated the same as an explicit request for html content.
Why is the mobile view not set for xhr requests? Since the first condition ensures that the request is for html content, it'd make sense to me that returned content would be different for desktop browsers and mobile devices.
I think this is for the same reason as above.
Would you be interested in a patch for this? Since these conditions seem to be so deliberated, I don't know if you'd consider this issue a defect.

Thank you in advance for any light you can shed about this.

—
Reply to this email directly or view it on GitHub.

[benlangfeld]

Sorry for the noise here, fat-fingering my phone. I'd certainly accept a proposal for a change to these rules, particularly easily if it were made optional without changing the default.

[aredondo]

Thank you for your prompt response.

What I can see in the Rails code is that when the request specifies Accept: */*, Rails simply loads formats with all available formats. This is done in ActionView::LookupContext. In a before filter—which is how set_mobile_format is executed—, it would therefore not be possible to know which format would be rendered. It is at render time that this decision is made depending on the available templates.

The simplest solution of all would be to remove the html format requirement altogether. Up to a certain point, this would be coherent with the fact that Mobile Fu does not even allow for a fall-back to html when a mobile template is not present. It's up to the programmer to enable Mobile Fu for specific controllers and/or actions.

I guess how suitable this solution is depends on whether there is a foreseeable situation in which a single route could return mobile and desktop html templates, and also json data or in other format. In my specific case there are routes that could return json or html data—but then, for these routes the html format does not depend on whether the request is made by a mobile device, so Mobile Fu is not needed.

What do you make of this? Do you have some other type of solution in mind?

[aredondo]

For the moment, I've gone with the simplest solution and made this patch: aredondo/mobile-fu@ba31cb3. So far, I can confirm that with Google Webmaster Tools it's now possible to retrieve the mobile version of the site.

[sdarwin]

We also needed to add the above mentioned patch from aredondo: aredondo@ba31cb3

Except for a slightly different reason. CDN caching.

If the hypothesis is that "Google crawler is presented with desktop site" (which frequently occurs) this may cause the desktop site to be cached by the CDN. Subsequent requests from mobile devices retrieve the cached desktop page.

Can the patch be merged? It would be a good idea.