How to limit bot crawl rates
-
Yesterday Hostgator suddenly shut down my site completely. It took me 6 hours of phone calls and email and complying with their requests to get it opened up again. The reason: Bingbots had been crawling my site excessively putting server resources on overload. One hostgator rep told me that Bing related shutdowns have happened quite a few times, whereas Google does not allow it’s crawlers to get out of hand.
Now I am wondering what would be a sane number to limit the pageviews of bots to prevent a recurrence of this shutdown. Under the Wordfence advanced options, firewall rules, a bot’s pageviews can the throttled from 1 time per minute to 1920 times per minute. Which number would still allow Bing to index my pages without putting my server resources at risk? Anyone know?
The topic ‘How to limit bot crawl rates’ is closed to new replies.