iThemes Blocked Crawler

  • Baloydi Lloydi

    (@lordzden)


    9 years, 8 months ago

    Sorry for the delay in response.
    It looks like your site ‘blocks’ our crawler (and being more technical it blocks our user agent or IPs). When accessing the URL using a browser — we get a valid response.
    But when our crawler tries to fetch this it is being blocked (and sends back response code 500, which indicates an error)
    In order to resolve this, could you please allow our user agent which is “Mozilla/5.0 (Java) outbrain” .
    Maybe you also block our IPs.
    These are the address ranges of our data centers, out of which we execute crawling.
    4.28.73.128/27 [4.28.73.128 → 4.28.73.159]
    74.201.198.64/27 [74.201.198.64 → 74.201.198.95]
    204.145.89.32/27 [204.145.89.32 → 204.145.89.63]
    64.94.43.96/27 [64.94.43.96 → 64.94.43.127]
    74.217.148.96/27 [74.217.148.96 → 74.217.148.127]
    4.71.33.160/27 [4.71.33.160 → 4.71.33.191]
    50.31.185.32/27 [50.31.185.32 → 50.31.185.63]
    64.74.232.32/27 [64.74.232.32 → 64.74.232.63]

    Please help! Thanks

    https://wordpress.org/plugins/better-wp-security/

Viewing 2 replies - 1 through 2 (of 2 total)
  • iThemes Support

    (@ithemes-support)

    9 years, 8 months ago

    Hey Lordzden,

    Thanks for reporting this! I’ll make sure the developer sees this. I’ll report back here if I have any new information.

    Thanks,

    Gerroald

    Chris Wiegman

    (@chriswiegman)

    9 years, 8 months ago

    Hi Lordzden,

    I do believe the issue is the “Java” user agent in the hack repair.com blacklist. Can you try removing the following line from your .htaccess and let me know if it fixes it?

    Also, what is the crawler? I’m willing to remove this line permanently from the plugin but I want to know what it is that I’m doing it for first.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘iThemes Blocked Crawler’ is closed to new replies.