503 for Googlebot

Hello bydlo!
It is not always like that. As a first step could you uncheck “Immediately block fake Google crawlers” under “Firewall rules” in Wordfence options and see if that fixes it?

Great it seems like you have narrowed it down then. I have tested against a few installs with the tools you provided and have not been able to reproduce the problem. The bot simulator gets blocked at Security level 4 but that’s normal because that security level is high and the request is in practice a fake google bot.

I suggest you keep Wordfence at Security level 2 unless you are under attack. You can also tweak your Wordfence settings under “Firewall rules” to be appropriate for the type of traffic you have or want to have on your site.

Hello again axel!
Security level 2 is default for the Wordfence install so most users will likely be using that. However, we appreciate your input and I’m glad you were able to find an option that works for you.

@bydlo First, Wordfence provides support here to help users like you. It’s a cost we bear, provided free to you. At no time will cursing or personal attacks be allowed. As a free user you can uninstall the plugin at any time you feel you are not being supported at the level you deserve.

Second, if I understand the problem correctly, you are concerned that we are blocking google bots when the security level is set to 3? Since we obviously would be in a lot of trouble if we natively blocked google, we actually whitelist the IPs they provide. However, google changes IPs from time to time without saying much, so much so that we are probably going to remove the ‘block fake google bots’ option entirely. Plus tools like you were using that pretend to be a googlebot to see what the response is served get blocked because they are, well, fake and they cause confusion for people who aren’t as well versed as you are in how bots, crawlers, etc work.

If you are concerned about a 503 error being served as opposed to another code, I think 503 is the correct error to send since a search engine shouldn’t penalize you for it. It sees the content is there and hasn’t moved but isn’t available at the moment. However, since we whitelist Google as much as one can, it shouldn’t be blocked or throttled at all. On one of our test sites I had the security level set as high as it would go and still was able to fetch as google. This showed the site as viewable at any of the security levels I tried.

If you’d like to restate the problem and the solution you think would be optimal, without the expletives or accusations of lack of support in a free forum, I’ll be happy to investigate and answer the question further.

tim