Title: Blocking Robots
Last modified: August 22, 2016

---

# Blocking Robots

 *  [DanielNest](https://wordpress.org/support/users/danielnest/)
 * (@danielnest)
 * [11 years, 4 months ago](https://wordpress.org/support/topic/blocking-robots/)
 * Hi,
 * I have been informed by my hosting provider (SiteGround) that my CPU load was
   extremely high this month. After further investigation, they informed me that
   it’s mainly due to being aggressively crawled by robots/crawlers.
 * They advised me to use robots.txt to block all robots using:
 * User-agent: *
    Disallow: /
 * And then allow access to “whitelisted” robots, like:
 * User-agent: Googlebot
    Allow:/
 * Is that a wise move? Would I not potentially be disabling some useful bots as
   well? Is there a good “whitelist” of robots that it’s recommended to allow through?
   Is there perhaps another way to solve this, such as blocking only known “bad”
   robots? Maybe there’s a plugin that does that?
 * My website is: [http://nest-expressed.com/](http://nest-expressed.com/)
 * Any help is much appreciated!
 * Thanks,
    Daniel

Viewing 3 replies - 1 through 3 (of 3 total)

 *  [Justin Greer](https://wordpress.org/support/users/justingreerbbi/)
 * (@justingreerbbi)
 * [11 years, 4 months ago](https://wordpress.org/support/topic/blocking-robots/#post-5528797)
 * Blocking robots and the effect is relative to how much you care about search 
   engines. Your robot text above only tells search engines what you would like 
   then to do.
 * Your code is simple suggesting what search bots do, not preventing them. Crawlers
   can ignore the robots.txt file (legit ones will honor your robots.txt file). 
   Google and other search engines do not hog much CPU at all. What you are in need
   of is a bad bots ban list that is implemented by .htaccess. You can put the code
   in yourself but they change all the time.
 * Once you have the ban in place, you will notice that you may still be hit a bit,
   this will be normal as the bots will continue to head butt your server until 
   it realizes that it is wasting its time. Them they will back off. The best plugin
   with support for bad bots i have seen is the “Better Security” plugin. In my 
   opinion it is the best an it has been battle tested on all my servers with great
   success.
 * Example if bad bots in htaccess: [http://www.javascriptkit.com/howto/htaccess13.shtml](http://www.javascriptkit.com/howto/htaccess13.shtml)
 * Better Security Plugin: [https://wordpress.org/plugins/better-wp-security/](https://wordpress.org/plugins/better-wp-security/)
 * Edited:
    The bad bots that you speak of are trying to attack your site, not crawl
   it for search engines. They are programmed to look for weakness and in the process
   they will kill your site by hogging all connections and CPU.
 *  Thread Starter [DanielNest](https://wordpress.org/support/users/danielnest/)
 * (@danielnest)
 * [11 years, 4 months ago](https://wordpress.org/support/topic/blocking-robots/#post-5528799)
 * Hi Justin,
 * Thank you so much for such a thorough response. I shall try the plugin, since
   it seems to be the easiest and most complete way to deal with this!
 * Best, Daniel
 *  [Justin Greer](https://wordpress.org/support/users/justingreerbbi/)
 * (@justingreerbbi)
 * [11 years, 4 months ago](https://wordpress.org/support/topic/blocking-robots/#post-5528808)
 * 😉

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Blocking Robots’ is closed to new replies.

## Tags

 * [google bot](https://wordpress.org/support/topic-tag/google-bot/)
 * [robots.txt](https://wordpress.org/support/topic-tag/robots-txt/)

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 3 replies
 * 2 participants
 * Last reply from: [Justin Greer](https://wordpress.org/support/users/justingreerbbi/)
 * Last activity: [11 years, 4 months ago](https://wordpress.org/support/topic/blocking-robots/#post-5528808)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics