Help with appending to htaccess

I haven’t tested this and you would need to change it to add the bad bots instead of the good ones you could also list by ip’s you need to include the [OR] on each RewriteCond you list except the last one. Back up the good htaccess file and add these lines (after the necessary edits) before the WordPress section. You can test it by trying your computer’s ip on the first REMOTE_HOST line to see the effect.

RewriteEngine On
RewriteCond %{REMOTE_HOST} 122\.122\.122\.122 [OR]
RewriteCond %{REMOTE_HOST} 123\.122\.122\.122 [OR]
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule .* http://%{REMOTE_ADDR [L]

This approach takes almost all the load off your server, as it never allows the remote to access WordPress

You can get a list of IP’s and user agents to block from the report dashboard in Wordfence if you install that.

bcr8tive, I’ve had my fair share of bad bots recently too…all part of brute force attacks on my websites.

For starters, mikeyarce mentioned a good plugin for security. I’ve been using it and Better WP Security.

Better WP Security helps you to get started blocking bad bots by including the list accumulated from HackerRepair.com. The way that BWPS puts them into your .htaccess is like this:

# Begin HackRepair.com Blacklist
RewriteEngine on
# Abuse Agent Blocking
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bolt\ 0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot\@yahoo\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CazoodleBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Default\ Browser\ 0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} discobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ecxi [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GT::WWW [NC,OR]
RewriteCond %{HTTP_USER_AGENT} heritrix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HTTP::Lite [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} IDBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} id-search [NC,OR]
RewriteCond %{HTTP_USER_AGENT} id-search\.org [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ISC\ Systems\ iRc\ Search\ 2\.1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Java [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Link [NC,OR]
RewriteCond %{HTTP_USER_AGENT} LinksManager.com_bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} linkwalker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} lwp-trivial [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Maxthon$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} MFC_Tear_Sample [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^microsoft\.url [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Microsoft\ URL\ Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Missigua\ Locator [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla\.*Indy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla\.*NEWT [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Nutch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} panscient.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PECL::HTTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PeoplePal [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PHPCrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PleaseCrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Rippers\ 0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SeaMonkey$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck\.internetseer\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Snoopy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Steeler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Toata\ dragostea\ mea\ pentru\ diavola [NC,OR]
RewriteCond %{HTTP_USER_AGENT} URI::Fetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} urllib [NC,OR]
RewriteCond %{HTTP_USER_AGENT} User-Agent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web\ Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} webalta [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebCollage [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Wells\ Search\ II [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WEP\ Search [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WWW-Mechanize [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zermelo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus\.*Webster [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ZyBorg [NC]
RewriteRule ^.* - [F,L]
# Abuse bot blocking rule end
# End HackRepair.com Blacklist

So, you can use this list and it gives you an example of syntax if you need to add to your .htaccess.

The next thing you can do is to use the IP address of the offending user and list it like this:

Order allow,deny
Allow from all
Deny from 101.66.88.202
Deny from XXX.XXX.XXX.XXX

Each IP address would be on its own line and be written as “Deny from XXX.XXX.XXX.XXX (replace X’s with IP address).

Your best bet though is to use either Wordfence or Better WP Security, or both. They will help.

Hope this helps.

@robinintexas,

I’ve not seen anything about that RewriteRule. I’ve done some pretty wide searches, but don’t find anything current on it. Perhaps you have a source that I can look at?

Thanks

Hi bcr8tive,

I’m sorry that I missed your reference to WF… my bad. I use both WF and BWPS together and have not had problems with them conflicting. Lately, I’ve had to do a lot of .htaccess changes as well. One problem I seem to have stumbled upon with WF is the Advanced Blocking option. This is where you would enter in any UAs or IP ranges. I keep getting errors there. So, what I’ve had to do is manually make those additions.

I understand your concerns of BWPS and it’s need to hook into WP core. This was a big decision for me to make. My problem with BWPS is that I cannot add UAs into it.

I did look at that link for onextrapixel and it does have a lot of great tricks.

I am not real advanced when it comes to .htaccess, but I have had to use it a lot over the years and each time I get attacked by botnets I have to update my ban lists. Right now I seem to be getting a lot of activity from user-agent Mozilla/4.0. When I had it set in WF, it worked well for a couple of days, but unbeknownst to me, some browsers were getting errors.

In your first post, you said:

What I have researched needs to be added looks like the following:

order allow, deny

deny from _________.com

allow from all

But I need to know HOW to add this addendum correctly to the preexisting WordPress htaccess document.

My experience is to write like this:

Keep this undisturbed

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

For rewrites you would write similarly:

RewriteEngine on
# Abuse Agent Blocking
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bolt\ 0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot\@yahoo\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CazoodleBot [NC,OR]
RewriteRule ^.* - [F,L]

Each line is a different rewrite… do not wrap.

For Allow/Denies:

Order allow,deny
Allow from all
Deny from 101.66.88.202
Deny from 109.1.137.192
Deny from 110.85.105.49
Deny from 110.85.68.159
Deny from 112.111.186.191
Deny from 113.165.15.208

You want to have an allow from all then list the denies individually, one per line. This deny list could become long and in some circumstance you would have to deny from a string of IP addresses similarly to this:

Deny from 117.26.0.0/16
this denies access to IPs 117.26.0.0 thru 117.26.0.16

You also want to write rules to deny access to certain files:

<files .htaccess>
Order allow,deny
Deny from all
</files>

<files readme.html>
Order allow,deny
Deny from all
</files>

<files readme.txt>
Order allow,deny
Deny from all
</files>

<files install.php>
Order allow,deny
Deny from all
</files>

<files wp-config.php>
Order allow,deny
Deny from all
</files>

If there are ANY of these files that you want accessed by a certain person, then you would list their IP as allow. Although on these particular files, I see no reason for anyone access these files.

I made an unfortunate mistake yesterday by having too many of these:

RewriteRule ^(.*)$ - [F,L]

This is used like an ending tag to a rewrite rule. If you don’t have the beginning of that rule, then you basically forbid anyone from seeing your website.

The correct syntax is as I wrote above.

Anyway, I hope this answers some of your questions. As I said, I’m not an expert but I do understand enough to get me out of trouble if I fail. 😉

Just a few optimizations, if you don’t mind.

RewriteCond %{HTTP_REFERER} stuff [NC]
RewriteRule .* - [F,L]

Would be better without the L flag because it’s assumed with the F flag.

http://httpd.apache.org/docs/current/rewrite/flags.html#flag_f

“When using [F], an [L] is implied – that is, the response is returned immediately, and no further rules are evaluated.”

RewriteCond %{HTTP_REFERER} stuff [NC]
RewriteRule .* - [F]

Also, you could use mod_rewrite to block access to files, which would probably more efficient like so:

RewriteRule ^((install|wp-config)\.php|php5?\.ini|readme\.(html?|txt)|\..+)$ - [NC,F]

I included the blocking of readme.htm as well as any hidden file, not just .htaccess, but anything that begins with a literal period. I also included the possibility of php5.ini files, which some hosts allow.

You could probably do this with FilesMatch as well, something like this below:

<FilesMatch "^((install|wp-config)\.php|php5?\.ini|readme\.(html?|txt)|\..+)$">
Deny from all
</FilesMatch>

Yes, there are a lot of different ways to do things. The reason I suggest using mod_rewrite most of the time is because WordPress already uses it for Permalinks. Once you use mod_rewrite you should try to stick with it. Because when you start using other directives (Files, FilesMatch, Redirect, etc.) with mod_rewrite and they’re not positioned correctly it could cause problems or keeping some of the code from working.

Just use whatever works for you, but try to keep it as efficient as possible. I know some hosts that will suspend an account because their .htaccess hasn’t gotten quite inefficient or lengthy (of course they probably won’t tell the customer that). mod_rewrite is a bit more resource expensive than core directives like Files and/or FilesMatch, but you can offset that with REGEX. Sometimes it’s better to use Files or FilesMatch, it just really depends on the circumstance (like .htaccess files in sub-directories).

That’s how I started. And to this day, I’m continually learning better, more efficient ways to do things. 🙂

Another example would be to optimize the large blocking of User Agents. I use a different approach now. But if I was going to use the one suggested with BWPS, I would definitely use REGEX to combine some of them.

Before:

RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [NC,OR]

After:
RewriteCond %{HTTP_USER_AGENT} ^C(hinaClaw|usto) [NC,OR]

Before:

RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bolt\ 0 [NC,OR]

After:
RewriteCond %{HTTP_USER_AGENT} ^B(lackWidow|olt\s0) [NC,OR]

Also, I replaced the Bolt\ 0 with Bolt\s0 for security reasons. I could actually make it a bit more efficient using non-capturing grouping, but I don’t want to go there right now. 🙁

For the other user asking about blocking a certain provider of services that start with A. There’s really not a general User Agent or header that you can go by. You really have to go by IP or Host Name (you really don’t want to block anything in .htaccess using a Host Name, because it has to do a lookup for every single file and will be very resource expensive). And the thing about blocking all of it’s IPs is that some Kindle users, rely on it’s services, so you’ll block them as well. The only security script that handles this well, allowing exceptions to the blocking of A, is ZBBlock.