Support » Everything else WordPress » Bandwidth and bad_bots

  • I’ve read, read again and tried things, but to no avail, so I’ve scrapped my robots.txt and attempts at bot blocking with .htaccess
    My site is being eaten to the tune of over 100meg/day, the majority of that being bots. A couple of known search engines is fine, but bots from heaven knows where is daft, and I want them gone.
    I’ve read stuff at webmasterworld, diveintomark and more, but could anyone point me at something that is easy to setup / understand / use that will either block the scavengers or that I can easily configure ?
    I know a perfect solution does not exist, but seeing as all my own attempts have failed, any guidance would be gratefully recieved.

Viewing 5 replies - 1 through 5 (of 5 total)
  • php spider trap
    set that up. then in your robots.txt file, disallow ficticious directories and/or files. in your .htaccess file, point those ficticious directories and/or files to the spider trap file. so when a bot or visitor tries to access the ficticious diretory or file, their ip address will be banned.

    Thread Starter Mark (podz)

    (@podz)

    Many thanks – files uploaded .. I’ll post back any numbers.

    for those who are not subscribers of webmasterworld, a google search will point you to the same resource.

    Thread Starter Mark (podz)

    (@podz)

    I just checked it out by getting myself banned, and it works a treat 🙂

    andymerrett

    (@andymerrett)

    I now redirect all bad bots to a ‘formtrap’ page, which in PHP builds a form with 500 elements, but puts a 2000 second delay in between each creation! If the bot ever stayed that long without timing out, it would get trapped for over 27 hours!!

    My hope is it actually crashes the bot. It’s fairly light on resources too as hardly any page is generated. Since I installed it only one bot has come in. I don’t know if the others are trapped, or are now avoiding my site.

    Ha ha.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Bandwidth and bad_bots’ is closed to new replies.