FYI The site is not behind Cloudflare.
Not sure what’s happening here looking at the code. I wonder if Google uses a different type of server to check whether a URL can be indexed, prior to enable its bots on a site.
Thanks for this, I have passed this onto our development team for further investigation
Please can you supply the information which allowed you to come to your conclusion? You’ve only mentioned the final diagnosis, but not how you got there. N.B. It’s common for bad bots to pretend to be the GoogleBot, so if you’re just going on the user agent, that’s not enough.
I have not reached a definitive conclusion yet but here is what happened:
On 4/11, I helped a client update their website to a brand new one. It was almost a complete re-do with new theme and plugins. They had been using AIOS on the previous site without issues but hadn’t turned on all the possible security measures. Most likely I had gone in and turned on the ‘Internet Bots’ setting on the staging site before the transition.
Once the transition was done, all appeared well but the site disappeared from Google search results. The Google search console indicated ‘Page indexed with no content’. If I tried to trigger an index request, it would return as ‘Rejected’. This is why I am suggesting that there is a testing process that Google uses before indexing and maybe that’s what is getting blocked.
Trying to figure out what was wrong, I ran a site scan from the Microsoft Bing Webmaster tools and it was only scanned a single page, with no errors reported. Which meant it was scanning an empty page.
At some point, I suspected the site was getting blocking by a security feature. I then ran experiments with Chrome, changing the user agent to be Googlebot. Once I saw I got a blank page, I looked for the security settings that could explain that. The ‘Internet Bots’ setting explained the blank page in my test, though it explicitly said it would not blocked a legitimate Googlebot or Bing. I went ahead and deactivated the plugin. I was then able to scan the site from the Bing Webmaster tools.
Going back to the Google search console, I was able to successfully use the ‘Test Live URL’ feature and then trigger an indexing request. A few hours later the site was indexed by Google. So it’s pretty clear the plugin setting is at fault. Even though, looking at the code, it’s not obvious why. I unfortunately do not have access to the server logs to figure out what the server requests look like.
The site is a managed WordPress site on GoDaddy using the GoDaddy CDN.
You mentioned you have tried to set your Chome user agent to Google; try to visit your WP site, and It is giving a blank page. It is legitimate because the plugin logic prevents dummy google bots like it.
May I know whether you are using any caching plugins? I am asking because the cache plugin shows the cached result to non-logged-in users and the actual results to the logged-in result. Sometimes the cache plugin shows blank pages to non-logged-in users and legitimate Google and Bing bots. If you disable the cache plugin, Does the issue resolved?
May I know your site URL?
Understood about testing with a different user agent. That was only to figure out why Google/Bing were seeing blank content.
No caching plugin whatsoever. Only the GoDaddy CDN and that should have no effect, as crawling works when the plugin is disabled.
I kindly ask that you add a proper warning on that setting page. I’m not the only who has reported this issue and there is no easy way to properly test whether it’s blocking legitimate bots.
I’m not comfortable sharing the URL here as it belongs to a client. If you have another way for me to send it to you directly, let me know.
Hi,
I will make a note internally to look at putting warning in the plugin
