Support » Plugin: Wordfence Security - Firewall & Malware Scan » Google crawler blocked when Wordfence activated

  • Resolved MarkDenis

    (@markdenis)


    Hi, I got a message from Webmaster Tools that “Googlebot can’t access your site”
    So I deactivated all my plugins, fetched as Google and the fetch was complete.

    So with all my plugins deactivated I reactivated them one by one while testing Fetch As Google in Webmaster Tools after each change.
    When I reactivated Wordfence the fetch was returned Unreachable.
    I deactivated Wordfence and the fetch was complete.

    I would like to keep using Wordfence. Can you advise on whatever setting I need to change.

    Thank you

    https://wordpress.org/plugins/wordfence/

Viewing 12 replies - 1 through 12 (of 12 total)
  • Did you accidentally blocked GoogleBot? take a look to your htaccess

    Hi, I have not had a reply from Wordfence so changed to iThemes security plugin and deleted Wordfence. Access for Googlebot is fine now.

    I’m noticing the same problem, when I tell Google to crawl my sites and render a preview, the results are just a bunch of messed up code – this cannot be good for SEO. I didn’t catch this initially with some of my other sites, because I had set up Google Webmaster Tools long before I installed Wordfence.

    I will be removing this from all of my sites as well, if there aren’t any official responses on here sometime soon.

    I have no problem with Google crawling any of my sites.

    In my Options-Firewall rules, I have the following crawler settings:

    Immediately block fake Google crawlers: (unchecked)
    How should we treat Google’s crawlers: Verified Google crawlers have unlimited access to the site
    If anyone’s requests exceed: 60 per minute then throttle it.
    If a crawler’s page views exceed: 60 per minute then throttle it.

    I would imagine that on larger sites then the request/page views upper limit might need to be increased to allow for genuine crawlers.

    I have the same options and still wordsfence will not let google bots crawl

    I am having the same problem too and have had to turn off Wordfence. This has been a great plugin for me until recently.

    Me too. It sucks because I love this plugin and now have to get rid of it

    Me too. It really sucks. I even paid for the pro version.

    Plugin Author WFSupport

    (@wfsupport)

    Are any of you guys blocking the US for pro version? If so, Google will not be able to crawl your site, nor will Bing or Yahoo.

    @tomhouy That sounds like an issue with multiple caching going on. Do you have ours and someone elses running? If two are using gzip browser caching at the same time, the browser sees the gzip compression and decompresses it once, resulting what you see.

    @ryan Can you turn off everything but wordfence and see if you are still blocking google? I’d like to see your htaccess file as well.

    @anne Same thing as Ryan but can you post in the premium forums so I can assist you there? If you prefer the free ones, then open a post just for your issue and post the link here for me so I see it.

    @everyone I totally understand posting similar issues together. I appreciate being able to see that a larger problem exists outside of one person’s installation. However, the wordpress forum rules ask you post separate issues. So if you guys can post separate issues and then comment on the similar ones it would avoid the mods coming in. Thanks.

    tim

    Hi all,

    I did as Tim said above and unblocked USA. I was then able to fetch as google and render the site.

    Thanks Tim!

    Aisling

    Plugin Author WFSupport

    (@wfsupport)

    Not a problem 🙂

Viewing 12 replies - 1 through 12 (of 12 total)
  • The topic ‘Google crawler blocked when Wordfence activated’ is closed to new replies.