Hi Sam,
Do you know if some of the sites account for a lot more of activity and the calls to noc4.wordfence.com? One call per minute is quite a lot — this is for updating the firewall rules and related data, and reporting attack data. During attacks, it can happen more often, but they usually come in waves, and I wouldn’t expect them to be nearly continuous.
The wordfence_syncAttackData calls also occur during attacks. It’s usually ok if these take a bit longer, since they typically run in the background, not while a user is waiting for a page to load. (It syncs data between a file on disk and the database, since the database is not available when the firewall runs before WordPress has loaded.)
That is quite a high number in P3 Plugin Profiler, and it is rare for that to happen. If it happens on all of the sites, or on a significant number of sites with different plugins, it may be related to I/O. Do the sites share file storage using NFS or similar?
-Matt R
Hi Matt,
Thanks for coming back to me. My guess is that between the 40 plus sites, they probably are pretty much constantly under attack between them. I can see wordfence_syncAttackData has been called around 6,000 times on our servers (I can see this from running queries on Apache logs) in the last 12 hours, with most of the calls coming from a dozen sites, that have presumably been attacked more than the rest.
We don’t use NFS, but I do see high IO on these web servers, a reasonable amount of which coming from Apache (looking at iotop), so I think the frequent writing to disk could be causing an issue for us. It’s maybe compounded by the fact we have a load balanced master/slave web server setup running where rsync is updating the slave with all file changes in the websites.
I would prefer not to turn the Wordfence firewall off on these sites – is there any way to turn the writing to disk and synching off without turning off the firewall?
Thanks,
Sam
Hi again Matt,
I’ve looked a little further into this and found the following:
1) The IO was a red herring I think – an IO monitor (New Relic) was telling me IO was 100%, but turned out to be because of an unfinished transaction somewhere – once I rebooted the server, that sorted itself out and IO on the servers is actually fine. I have excluded wflogs files from rsyncing between servers now anyway, as I think this might have been confusing matters and is unecessary work. Out of curiosity, do you know what would happen if the same site has two different copies of the wflogs files on different load balanced servers? (since different requests will come to each server)
2) I don’t think noc4.wordfence.com is causing a problem for website visitors on my server. I’m currently seeing around 30 calls per minute to it between all the sites and it is consistently the most called, slowest external service, but so long as it runs in the background and doesn’t eat too many server resources, I don’t suppose it should be an issue.
3) Most importantly, I think I’ve finally figured where the main issue is that has been slowing down the response to website visitors. Looks like it simply came down to /wp-content/plugins/wordfence/tmp and /wp-content/wflogs folders not having permission for the web server account to write to them. I could see Wordfence was trying to chmod them and getting permission denied on a rename relating to them and I guess this was taking several seconds for some reason. I changed these folders from 755 to 775 on all our sites and now this appears to be resolved. I could see the tmp folder not having correct permissions showed up in the Wordfence diagnostics page, but it certainly wasn’t clear this was causing such a major issue, so it might be worth putting something more prominent in, in case anyone else has this issue and doesn’t realise.
Thanks,
Sam
Hi Sam,
Sorry for the late reply. I do think it will generally be ok to have the wflogs files not rsyncing between the servers, even though it isn’t ideal. It may mean that one of the servers will have outdated firewall rules for some time, but it will likely correct itself periodically. You might lose some of the blocked hits in the Live Traffic view, as well. I think the worst case is if you need to change settings on the Firewall page, like turning the firewall on/off or whitelisting a false positive, the changes will only affect the server you’re on at the time — you could always temporarily re-enable the rsync if you need to make changes (that should generally be rare.)
Thanks for the information on the file permissions — I’m glad you got it working better. I haven’t seen that occurring on any of our test servers when looking at permissions issues before, but we’ll check it out more. It might not be easily reproduced, but we can try making the notice more prominent if we can’t cause the same issue on a test site.
-Matt R
No worries and thanks for coming back Matt. We generally turn off the live traffic view, so that element’s not a problem. Performance seems to have improved since excluding synching on wflogs – I guess that comes at the expense of having well tuned firewall rules, although haven’t had any complaints on that score yet (i.e. visitors or WP users receiving 503s). We’re planning to install the Apache Mod Sec WAF commercial rules from Trustwave so we can maintain rules in one place that run as soon as users hit Apache for any of our sites (including some non WP sites), so may turn off the Wordfence WAF to save doubling up. Out of curiosity, have you heard of any clashes between Apache based WAF and Wordfence WAF? I presume they would work okay together, but it’s just a bit redundant to have the server use resources carrying out a similar set of checks twice on each request.
The file permissions issue made a huge difference for us, but maybe there’s something unusual about the setup we have. All is working fine now those folders have the right permissions anyway.
Thanks,
Sam
Hi Sam,
I haven’t heard of any conflicts of running a WAF in both Apache and the Wordfence Firewall; we’ve done some testing here and haven’t had issues running both either. If you have any potential false positives that come up, just remember to check both mod_security’s log (assuming logging is enabled) and the Wordfence Live Traffic page.
-Matt R
