Yes, that is definitely a problem. A few other people seem to have had this problem and the dev team is investigating, but I do not know the cause yet. In two cases, I could see that Googlebot’s IP could not be resolved from certain hosts, but worked fine for others, both when using DNS tools outside of WordPress. (Google’s own documentation suggests using DNS reverse lookups to verify Googlebot IPs, which is what we do.)
Do you see any of these IPs in your current blocked IP list? If so, do you see the reason listed for the block, and the hostname listed, if any?
Or did you get any alert emails about the blocks, which show the reason?
In the meantime, we recommend changing these two options, and possibly making the other firewall rules less strict, depending on the “reason” listed in the blocked message:
Disable “Immediately block fake Google crawlers”
Set “How should we treat Google’s crawlers” to either “Anyone claiming to be Google has unlimited access to the site” or “Treat Google like any other crawler” (depending on your preference)
-Matt R
Thread Starter
ACEkin
(@acekin)
I will make the suggested settings changes Matt, I hope you and your team can get to the bottom of this as the blocking seems excessive.
I do not see these IP numbers on my blocked IP list. That is somewhat comforting, but still very concerning. In my settings Immediately block fake Google crawlers was not checked, I changed the other setting to Anyone claiming to be Google has unlimited access.
I did not receive any alert e-mails, but I did receive excessive “Server errors” warning from Google, error 503s. I am writing in case that is relevant.
Thanks
Cemal
Thank you for the additional details — can you also tell me which hosting provider you are using? We’re trying to narrow down possible causes, since the problem is not happening on any of our sites.
Can you also tell me what you have in the option “How does Wordfence get IPs”, in the top section of the options page? If you not using the CloudFlare CF-Connecting-IP option, do you have the CloudFlare plugin installed? (CloudFlare may not be relevant here, but this will help us compare to the few other sites with a similar problem.)
-Matt R
Thread Starter
ACEkin
(@acekin)
Mark, I sent you an e-mail with the screen capture of all my Wordfence settings. My provideer is DailyRazor.com. Wordfence gets the IP’s via CloudFlare and I have the CF plugin installed. Let me know if I can provide other information and check your mail for the email.
Cemal
Thanks for the email and other details — can you also tell me if you have anything blocked on the “Advanced Blocking” page?
-Matt R
Thread Starter
ACEkin
(@acekin)
Yes I do, the screen capture is coming via e-mail.
Cemal
