Title: Bots
Last modified: August 19, 2016

---

# Bots

 *  [giantman](https://wordpress.org/support/users/giantman/)
 * (@giantman)
 * [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/)
 * In my robots.txt I have:
 * User-agent: *
    Disallow: /images/
 * I then use teleport pro webcrawler to “gather” my site and it is still retrieving
   the images inside of /images/ folder.
 * Is there a way to block ALL bots using .htaccess EXCEPT for the googlebot? I’m
   showing tons of spiders crawling my site in my logs. I think a lot of them don’t
   obey the robots.txt

Viewing 5 replies - 1 through 5 (of 5 total)

 *  Thread Starter [giantman](https://wordpress.org/support/users/giantman/)
 * (@giantman)
 * [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/#post-1758859)
 * WTF??? I just put:
 * User-agent: *
    Disallow: /
 * in robots.txt and teleport Pro is still finding the images and other files.
 *  Thread Starter [giantman](https://wordpress.org/support/users/giantman/)
 * (@giantman)
 * [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/#post-1758860)
 * SetEnvIfNoCase user-agent “^Teleport\ Pro” bad_bot=1
 * <FilesMatch “(.*)”>
    Order Allow,Deny Allow from all Deny from env=bad_bot </
   FilesMatch>
 * Just tried that, and teleport pro is STILL finds the images and 60 other files.
   grr.
 *  Thread Starter [giantman](https://wordpress.org/support/users/giantman/)
 * (@giantman)
 * [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/#post-1758862)
 * WTF????????????????
 * I put:
 * Options -Indexes
 * into an htaccess file inside /images/ and teleport pro is STILL finding the images.
   WTF!!!!!!!!!!!!!???????
 *  [Clayton James](https://wordpress.org/support/users/claytonjames/)
 * (@claytonjames)
 * [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/#post-1758867)
 * > Options -Indexes
 * That just limits casual browsing of your directories in the absence of an index
   file.
 * Google might have some re-write tips for you.
 * [“how to” ban “teleport pro” teleport bot](http://www.google.com/#sclient=psy&hl=en&q=%22how+to%22+ban+%22teleport+pro%22+teleport+bot&aq=f&aqi=&aql=&oq=&gs_rfai=&pbx=1&fp=6e8733203d1b4e27)
 * This might spark some ideas or lead to more helpful information –[ An example WordPress robots.txt file: ](http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress)
 *  Thread Starter [giantman](https://wordpress.org/support/users/giantman/)
 * (@giantman)
 * [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/#post-1758870)
 * dang…I just downloaded “blackwidow” webcrawler from download.com and its crawling
   my whole site finding almost everything…and I have about 6 security plugins blocking
   all kinds of stuff, plus a huge htaccess file that is supposed to be blocking
   BlackWidow and 100’s of other bots….and it’s still crawling my site….
 * This is bad….

Viewing 5 replies - 1 through 5 (of 5 total)

The topic ‘Bots’ is closed to new replies.

 * In: [Everything else WordPress](https://wordpress.org/support/forum/miscellaneous/)
 * 5 replies
 * 2 participants
 * Last reply from: [giantman](https://wordpress.org/support/users/giantman/)
 * Last activity: [15 years, 5 months ago](https://wordpress.org/support/topic/bots-1/#post-1758870)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics