• Yesterday Hostgator suddenly shut down my site completely. It took me 6 hours of phone calls and email and complying with their requests to get it opened up again. The reason: Bingbots had been crawling my site excessively putting server resources on overload. One hostgator rep told me that Bing related shutdowns have happened quite a few times, whereas Google does not allow it’s crawlers to get out of hand.

    Now I am wondering what would be a sane number to limit the pageviews of bots to prevent a recurrence of this shutdown. Under the Wordfence advanced options, firewall rules, a bot’s pageviews can the throttled from 1 time per minute to 1920 times per minute. Which number would still allow Bing to index my pages without putting my server resources at risk? Anyone know?

Viewing 1 replies (of 1 total)
Viewing 1 replies (of 1 total)

The topic ‘How to limit bot crawl rates’ is closed to new replies.