Using Rate Limiting and Blocking Bots

  • Resolved daftshadow

    (@daftshadow)


    4 years, 6 months ago

    I recently started using the rate limiting feature to stop bots from overly crawling my sites. I was having a problem specifically with Amazonbot. My sites were getting thousands of requests from Amazonbot which caused performance problems for my sites. I don’t sell anything on Amazon. Is there any point to having Amazonbot crawl my sites? I already blocked it in robots.txt. It appears that Amazonbot follows this rule.

    This is currently my settings for rate limiting. I’m just a beginner when it comes to setting this up as I never used it before. Are these settings too strict? Or is it optimal? I followed the Wordfence guide on setting up rate limiting. It seems 240 per minute is the recommended value but what exactly does it mean? Compared to say 120?

    How should we treat Google’s crawlers – Verified Google crawlers will not be rate-limited
    If anyone’s requests exceed – 240 per minute, then throttle it
    If a crawler’s page views exceed – 120 per minute, the throttle it
    If a crawler’s pages not found (404s) exceed – 60 per minute, then throttle it
    If a human’s page views exceed – 240 per minute, then throttle it
    If a human’s pages not found (404s) exceed – 60 per minute, then throttle it
    How long is an IP address blocked when it breaks a rule – 30 minutes

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support wfphil

    (@wfphil)

    4 years, 6 months ago

    Hi @daftshadow

    Amazon state that their bot respects robot.txt files:

    https://developer.amazon.com/support/amazonbot

    As you say, it should no longer be crawling your site.

    You have set the option If a crawler’s page views exceed to 120 requests per minute. As stated in our documentation we recommend that you set this to 240 requests per minute. We also state that a search engine like Bing can send this many requests so your current setting can and may be blocking friendly bots:

    https://www.wordfence.com/help/firewall/rate-limiting/

    Thread Starter daftshadow

    (@daftshadow)

    4 years, 6 months ago

    @wfphil Thanks for the clarification. What about the other values that I have set to 60 requests per minute? Should I just changed all to 240?

    Also, is there any negative impact on my sites from blocking Amazonbot? I’m not exactly sure why it was crawling my sites. I don’t sell anything on Amazon. In fact my sites do not sell anything at all.

    Plugin Support wfphil

    (@wfphil)

    4 years, 6 months ago

    Hi @daftshadow

    Your other settings for 60 requests per minute are okay as long as you have understood the caveats.

    You will notice in the Amazon link in my last reply that they don’t mention the marketplace, they mention Alexa instead. At the bottom of their page there is a contact email if you have questions or concerns so they can answer any questions that you have.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Using Rate Limiting and Blocking Bots’ is closed to new replies.