User-agent blocking doesn't seem to be working

Michael Fraase
(@mfraase)

11 years, 5 months ago

Thanks kindly for adding user-agent blocking — what a great addition to an already great service. For some reason, user-agent blocking doesn’t appear to be working for me.

WHAT I DID:
Added a bunch of rogue bots to Wordfence’s Advanced Blocking page like so:
IP address range: empty
User-Agent (browser) that matches: baidu*, seokicks*, yandex*, smelt* (and a bunch more without wildcard patterns)
Referer (website visitor arrived from) that matches: empty
Enter a reason you’re blocking this visitor pattern: Bad bots

WHAT I EXPECTED:
A multitude of blocked hits reported next to each entry on the Advanced Blocking page.

WHAT HAPPENED:
Word fence’s Advanced Blocking page reports
0 blocked hits
Last blocked: never

When I go to Wordfence’s Live Traffic page, I can sit and watch entries like this roll by all day long:

Germany Germany left http://www.farces.com/archives/timeline/ and visited http://www.farces.com/robert-hunter-dr-john-to-receive-americana-lifetime-achievement-award/
41 seconds ago IP: 144.76.182.139 [block] Hostname: c1.seokicks.de
Browser: SEOkicks Robot version 0.0
Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)

Please advise.

https://wordpress.org/plugins/wordfence/

Viewing 11 replies - 1 through 11 (of 11 total)

asaracena
(@asaracena)

11 years, 5 months ago

I’m having a similar issue. November 13th all of my WordPress sites had unauthorized users added and then some of these were able to successfully login. All of these successful logins were coming from an address like this:
bb25194c.virtua.com.br

So I blocked this site and any permutations using a wildcard:
*virtua.com.br

Today I had this message:
A user with IP address 179.234.234.98 has been locked out from the signing in or using the password recovery form for the following reason: Exceeded the maximum number of login failures which is: 3. The last username they tried to sign in with was: ‘Admin’
User IP: 179.234.234.98
User hostname: b3eaea62.virtua.com.br

In Advanced Blocking it says this:
IP Range: Allow all IP addresses
Browser Pattern: Allow all browsers
Source website: Block visitors from websites that match the pattern: *virtua.com.br
Reason: known hacker
0 blocked hits
Last blocked: Never

The firewall is enabled so I don’t understand why this user wasn’t blocked immediately, not after 3 tries.

Is this working? Or am I doing something wrong?

Thread Starter Michael Fraase
(@mfraase)

11 years, 5 months ago

asaracena, your problem has nothing to do with user-agent blocking. Kindly start your own support thread.

The “Source website:” field in Wordfence’s Advanced Blocking screen refers to the Referer (website visitor arrived from) that matches: field. It’s for what it says: Referers.

If you want to block entire domains, you’ll need to do it in your .htaccess file like so (assuming Apache 2.4):

# BEGIN entire domain blocks
<RequireAll>
Require all granted
Require not host .virtua.com.br
</RequireAll>
# END entire domain blocks

If you’re running Apache 2.2, you’ll need to use something like this instead:

# BEGIN entire domain blocks
<Limit GET POST PUT>
Order Allow,Deny
Allow from all
Deny from .virtua.com.br
</Limit>
# END entire domain blocks

WFSupport
(@wfsupport)

11 years, 5 months ago

@michael Thanks for helping the other user 🙂

Is your firewall options enabled on the options page?

tim

Thread Starter Michael Fraase
(@mfraase)

11 years, 5 months ago

Thanks for the response.

Yup, firewall options are fully loaded.

The only odd thing about my situation is that I’m running OS X 10.10.1/Server 4.0. That includes PHP 5.5.x and Apache 2.4.x.

WFSupport
(@wfsupport)

11 years, 5 months ago

First, let me assure you that no one here is a mac hater. 3/4 of the company is on macs.
Second, I don’t know that we officially support Mac Server OS. Let me get to the dev team for suggestions and get back to you.

Unless, of course, anyone in the community here has run on a mac os before?

tim

WFSupport
(@wfsupport)

11 years, 5 months ago

Can you open your java console in chrome (settings menu > more tools > javascript console) and see if there are any red errors showing?

tim

Thread Starter Michael Fraase
(@mfraase)

11 years, 5 months ago

OK, with the disclaimer that I’m not a coder, I launched Chrome Canary, navigated to my home page, and opened View > Developer > JavaScript Console.

Nothing much of interest there and the only errors I see are strikethroughs for some of my typography overrides in CSS.

Similarly, nothing jumps out at me when I navigate to the appropriate Wordfence admin pages and look at the Javascript Console. But then, I don’t really know what I’m looking for.

If you can point me to a specific page and specific things to look for in the Chrome JavaScript Console, I’m happy to do it.

For what it’s worth, I consulted the Google and nosed around a little, finding this:
https://chrome.google.com/webstore/detail/javascript-errors-notifie/jafmfknfnkoekkdocjiaipcnmkklaajd

I installed it — it’s supposed to notify of any JavaScript errors and it reports none for my home page, admin page, Wordfence admin page, or Wordfence Advanced Blocking page.

Plugin Author Mark Maunder
(@mmaunder)

11 years, 5 months ago

Hi Michael,

I’m not sure what the problem is here. I don’t think it’s a javascript error because this is server side code that blocks visitors based on their user-agent and appears to not be working.

There are plugins that let you modify your user-agent. Can you try one of them and see if you can temporarily block yourself?

Also I noticed all the patterns you tried are a prefix followed by a wildcard e.g. yandex*

Please try a wildcard with the text in the middle like: *yandex* and see if that works. Perhaps it’s a problem specific to patterns that end with an asterisk – and you can test which is working by using that plugin that lets you change your own user agent.

Regards,

Mark.

Thread Starter Michael Fraase
(@mfraase)

11 years, 5 months ago

Hi Mark,

I’m thoroughly confused now and I don’t want to waste any more of your or your team’s time on this. After all, I’m using the free version of Wordfence and have no reasonable expectation of support. I suspect this may be related to something Apple did to its Apache 2.4 implementation. My .htaccess domain blocks that worked fine in Apache 2.2 cause all sorts of problems when converted to Apache 2.4 syntax.

At any rate, I added User-Agent Switcher for Chrome 1.0.38 to Chrome Canary and defined my user agent as the Baidu spider string (Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html). I accessed my home page, unblocked. Here’s a partial access_log entry:

http://www.farces.com 23.25.155.113 – – [25/Nov/2014:08:36:11 -0600] “GET / HTTP/1.1” 200 12263 “-” “Mozilla/5.0 (compatible; Baiduspider/2.0)”

The rest of the log entries for that session were expected 304s for unchanged cache hits.

Surprisingly, this session did not appear at all in the Wordfence Live Traffic page.
Thread Starter Michael Fraase
(@mfraase)

11 years, 5 months ago
Here’s an update:

The tl;dr version: I still don’t know if Wordfence is blocking user agents or not.

I spent a good bit of the day noodling around with PHP 5.5.x Zend OpCache + Wordfence Falcon Engine + Autoptimize to replace W3 Total Cache because it occurred to me that maybe, just maybe Wordfence’s new user-agent blocking feature might work better. In past noodling, I remember that Falcon wants to write all of its code to the .htaccess file.

In my case, I have WordPress set up in a subdirectory with its index.php in the document root, so Wordfence wants to write its .htaccess file to /wordpress rather than the root directory. Fine by me, I guess.

The good news is that in my particular case PHP 5.5.x Zend OpCache + Wordfence Falcon Engine + Autoptimize yields significantly better performance than PHP 5.5.x Zend OpCache + W3 Total Cache. Faster page loads, better Google PageSpeed/Yahoo YSlow scores, smaller page sizes, and fewer resources loaded.

The bad news is that I don’t know if Wordfence is blocking user-agents or not. I had to keep cleaning out the error_log between Apache restarts so I never saw one, but that doesn’t mean it’s not working.

Here’s the user-agent blocking code Wordfence wrote to the .htaccess file:
```
#Blocking code for browser pattern: *ahrefs*, *baidu*, *seokicks*, *sogou*, *yandex*
	SetEnvIf User-Agent .*ahrefs.*, .*baidu.*, .*seokicks.*, .*sogou.*, .*yandex.* WordfenceBadBrowser=1
	Deny from env=WordfenceBadBrowser
```
Seems reasonable enough and it looks like it *should* work to me.
Thread Starter Michael Fraase
(@mfraase)

11 years, 5 months ago
Yet another update:

The Wordfence user-agent blocking code Wordfence wrote to the .htaccess file is definitely *not* working at all. Using the example above, and looking at my latest access_log, my website is being hammered by these bots (for some reason seokicks has slacked off):
```
www.farces.com 188.165.15.81 - - [25/Nov/2014:07:50:44 -0600] "GET /tag/grover-norquist/ HTTP/1.1" 503 2231 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
www.farces.com 220.181.108.153 - - [25/Nov/2014:07:55:17 -0600] "GET /category/internet/page/29/ HTTP/1.1" 200 16237 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
www.farces.com 61.135.189.185 - - [25/Nov/2014:10:08:28 -0600] "GET /medicare-eliminating-epogen-profit-incentive-worked/ HTTP/1.1" 200 16197 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
www.farces.com 94.198.4.61 - - [26/Nov/2014:01:03:54 -0600] "GET /wordpress/wp-content/uploads/2012/11/dharmachakra-wheel-of-law.jpg HTTP/1.1" 200 55971 "http://yandex.ru/images/search?img_url=http%3A%2F%2Fwww.canorml.org%2Fmedical_marijuana%2Fwheel-of-law.jpg&uinfo=sw-1280-sh-1024-ww-1264-wh-876-pd-1-wp-5x4_1280x1024&_=1416985270374&viewport=wide&p=2&text=%D0%BA%D0%BE%D0%BB%D0%B5%D1%81%D0%BE%20%D0%B1%D1%83%D0%B4%D0%B4%D1%8B&noreask=1&pos=65&rpt=simage&lr=213" "Mozilla/5.0 (Windows NT 5.1; rv:33.0) Gecko/20100101 Firefox/33.0"
```

Viewing 11 replies - 1 through 11 (of 11 total)

The topic ‘User-agent blocking doesn't seem to be working’ is closed to new replies.