Blocking fake Googlebot

This may be a DNS issue on your host, as it sounds like the reverse lookup is not working correctly, but I am not positive yet — it is working correctly on our test servers, but we will investigate further too.

For now, you may need to temporarily set “How should we treat Google’s crawlers” to the second option, “Anyone claiming to be Google has unlimited access”, to make sure they can crawl the site properly. Disabling “Immediately block fake Google crawlers” may be necessary too.

If you have SSH access to the server, can you try running the command:
dig -x 66.249.67.161
and post the results here?

-Matt R

Thanks for checking that. Yesterday on one host for me, “dig -x” was returning no PTR records for the above Google addresses, and the “status” said SERVFAIL instead of NOERROR, but it cleared up about 4 hours after my initial reply above. I think it was a temporary DNS issue, but I can’t tell for sure — reverse lookups were working for other hosts at the time, on the same host.

If you have Live Traffic enabled in Wordfence, you should be able to see if Google is currently being identified correctly by looking at the “Google Crawlers” tab.

The dev team is looking into this, to add a fallback if reverse DNS lookups are not working, since that seems to be the most likely issue here. Our internal reference number for this is FB1034.

If Google crawlers still aren’t being identified today, let me know. If it does happen again, could you also run “dig -x” for the IP that has a problem, as soon as you notice it?

-Matt R

Ok, thanks. Let us know if it comes up again, and the dev team will still be looking at adding a fallback for future reverse lookup issues, mentioned above.

-Matt R

@anna: Thanks for the report, and sorry for the trouble. Was the ‘dig -x’ command run on the same host that had this problem, and was the Google IP the same as one that was blocked on that host? (The PTR record shows the expected result in your screenshot, but the TTL is 21599, which is the default of 6 days that Google uses, so the result wasn’t cached locally.)

Changing these two options should allow the crawlers through, on sites that are having this problem, for now — it may be that the bad results have been cached, and will likely clear up:
Set “How should we treat Google’s crawlers” to the second option, “Anyone claiming to be Google has unlimited access”
Disable “Immediately block fake Google crawlers”

If you can post screenshots of any of the blocked IPs as well, if you have any still blocked, I can verify that there isn’t a new issue causing this. (We haven’t seen this problem on our sites, and the one host where I saw the DNS error doesn’t have a web site — so I don’t have much else I can check on our end. The dev team is already looking at handling failed DNS lookups, in case that is the issue.)

-Matt R

Can you post the IPs that are showing up, and try “dig -x” on those IPs again, from the affected host(s)?

Also, do you still have the options set that I had listed above? They should stop the blocking from happening, if the host is having trouble with the reverse lookups, or other possible issues.

-Matt R

Yes, it should be only temporary, but if there are consistent problems with this on your host, the fix may have to wait for a future release of Wordfence.

If you can still try “dig -x” on the affected host for the affected IP, or a few of the other IPs you mentioned in the original post above, that might still help. If any of them come up with a missing “ANSWER SECTION” and/or a status other than “NOERROR”, it would help to confirm the issue.

If any of the Google IPs that you look up have issues, please copy and paste the entire output from dig, like you did before. (If they’re successful and the “answer section” has a googlebot.com address, then we don’t need the output.)

-Matt R