Question about blocking fake Google crawlers

  • Resolved tomdkat

    (@tomdkat)


    8 years, 11 months ago

    Hi! I’m running Wordfence 6.3.9 and have the “immediately block fake Google Crawlers” option enabled. This morning, I found this entry in the Wordfence “Live traffic” log:

    Japan Japan tried to access non-existent page http://www.{mysite}.com/web/wp-admin/setup-config.php?step=1
5/30/2017 7:32:49 AM (26 minutes ago)   IP: 202.218.229.168 [block]   Hostname: ns.t-web.co.jp
Browser: undefined
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

    This tells me Wordfence didn’t automatically block the request, despite the user agent string at least attempting to appear to be a Google bot. How can I confirm the “immediately block fake Google crawlers” function is working?

    Thanks in advance!

    Peace…

Viewing 4 replies - 1 through 4 (of 4 total)
  • bluebearmedia

    (@bluebearmedia)

    8 years, 11 months ago

    Someone from WF will chime in I’m sure, but the short answer is that there is no infallible way to differentiate fake google crawlers from real ones – some will always get through.

    Even with that in mind, reducing the number of fake crawlers is still much better than letting them ALL in!

    Thread Starter tomdkat

    (@tomdkat)

    8 years, 11 months ago

    Thanks for the reply. Yes, I agree that reducing the number of fake crawlers is a GOOD thing. 🙂 I don’t get many log messages about fake Google crawlers being blocked and given the period strange traffic I get from systems using Google proxy:

    https://wordpress.org/support/topic/question-about-traffic-from-google-proxy/

    I just want to make sure things are working as I hope they are.

    Thanks!

    wfalaa

    (@wfalaa)

    8 years, 11 months ago

    Hi tomdkat,
    Regarding “Immediately block fake Google crawlers” option, we had some issues with it last year, hence we were forced to decrease its effectiveness to prevent valid Google bots from getting blocked by mistake, that’s why sometimes fake Google bots may slip through, however, we have a ticket opened in our system (internal reference number: #FB3354) to improve this option functionality.

    Thanks.

    Thread Starter tomdkat

    (@tomdkat)

    8 years, 11 months ago

    Hi! Thanks for the information!

    Peace…

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Question about blocking fake Google crawlers’ is closed to new replies.