hausinteractive
Forum Replies Created
-
Thanks @hjogiupdraftplus. Just a note that the original problem reported here still persists through updates of AIOWPS and W3TC.
Hey @hjogiupdraftplus. We have an update: disk: enhanced is filling up our storage and preventing nightly backups from completing within 24 hours so we’ve had to disable disk:enhanced and go back to memcache even though it fails from time to time.
Does anyone need to get in touch with W3 Total Cache or have you been already?
We checked the logs and this problem started happening around August 29, 2024 if that helps line the bug up with a version number for AIOWPS and W3TC.
- This reply was modified 1 year, 2 months ago by hausinteractive.
Thanks @hjogiupdraftplus. For the time being we are running memcache caching for DB and objects to see if the problem is still triggered. Our page_cache is set to disk:enhanced until there is a fix for memcache.
Thanks, @hjogiupdraftplus. Were there any changes in 5.3.5 that affect how this function is handled? We’ll obviously be updating but would like to know if we should try re-enabling memcache again.
So far we are 2 weeks into disk:enhanced and have not seen the problem. If we’re not making changes I’ll probably wait another 2 weeks before agreeing that memcache is the culprit.
Thanks @hjogiupdraftplus. If you’re willing to install memcached I’d love to see what your testing reveals. I think we’ll be able to find that answer much quicker than the W3TC folks would.
I think we’ll survive for a while with disk caching, and it sounds like we might need to anyway if the problem ends up being with W3TC and their interaction with memcached.
Follow-up question: What about memcache makes you suspect it is the culprit?
Thanks @hjogiupdraftplus, we will switch to ‘disk enhanced’ and monitor progress. We would prefer not to run in this mode long term as memcache provides significant performance that we need.
Thanks @hjogiupdraftplus
Spam Prevention -> “Comment spam IP monitoring” is enabled with 1,805 blocked IPs. Those should be in
{db_prefix}_aiowps_permanent_blocktable and in AIOS should be showing in WP security > Dashboard > Permanent block listYes. We see 6 addresses in this table that attempted to access the site on the most recent date this problem occurred. None of the timestamps coincide with the 127.0.0.1 redirect, however.
W3Total cache do have Memcache enabled?
Yes.
Is there any proxy server installed where your own server IP might get blocked?
No.
Here according to me cache is cleared then > visit from blocked IP of home page might create cache to the redirect 127.0.0.1. and applied for all. But how is this done have to check in more details. As this should not be the general case.
This 127.0.0.1 redirect happened again yesterday. We were able to detect it quickly and cleared the site cache. As expected, the home page was restored immediately. This is not surprising, but we have confirmed that when the problem is triggered it is being cached.
Unfortunately our monitor only checks the site every N minutes, so alerts are usually delayed by a few minutes, meaning we don’t get exact timestamps of when the redirect is triggered.
You mentioned certain POST requests potentially being problematic. This IP address raises an eyebrow due to:
• It being a POST
• It happening around the time of the redirect trigger
• It generating odd response codes (302 for the first request, 411 for the second request)168.227.229.96 www.mediaplaynews.com - [18/Nov/2024:04:19:14 -0800] "POST /wp-comments-post.php HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"7 seconds later:
168.227.229.96 www.mediaplaynews.com - [18/Nov/2024:04:19:21 -0800] "POST /wp-comments-post.php HTTP/1.1" 411 239 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"This IP address is NOT in the
{db_prefix}_aiowps_permanent_blocktable.Thanks @hjogiupdraftplus,
Do you have the W3 Total cache plugin is installed?
We do. Due to the traffic to the site we can not disable this. We are planning to manually clear the cache next time we see this issue. If the locked down home page is cached it should become accessible right away.
127.0.0.1 for all site visitors front pages redirected can only due to cache + IP permanently blocked by AIOS either due to Spam comments
Spam Prevention -> Comment Spam -> “Detect spambots posting comments” is enabled AND
Spam Prevention -> “Comment spam IP monitoring” is enabled with 1,805 blocked IPs BUT
The website home page does not have a comment form so it should not be able to trigger an IP block AND
These settings do make any mention of redirecting bad IPs to 127.0.0.1
or due to Ban POST requests that have a blank user-agent and referer
Firewall -> Internet Bots -> “Ban POST Requests that have a blank user-agent and referrer” is NOT enabled and has never been enabled.
OR you have 404 detection have locked out the IP and after cache flush first time that blocked IP visiting the site.
Brute Force -> 404 Detection is NOT enabled and was never enabled. We have changed the 404 lockout redirect URL to 127.0.0.2 and left the 404 detection to OFF. This will help test to see if the feature is “active” even though it is not “enabled” in the admin. If it is still active even though it’s set to OFF we’ll see this new redirect URL in the logs.
If you can check the database
aiowps_login_lockdowndo have that IPs blocked at the same time it will have release time also as it do not permanently block IP but temp lockout IP access. WP security > Dashboard > Lockout IP list will show only during temp lockout those IPs not after release.We have checked this DB table and there is one IP block from earlier this year that does not coincide with any of our 127.0.0.1 redirect timestamps. (this lockout is not even on the same day as a 127.0.0.1 redirect outage)
Thanks!
Thanks @hjogiupdraftplus. Answers:
Can you please cross-check the WP security > Dashboard > Permanent blocklist. Is there any IP blocked? If yes do there is any cache plugin on which may cache the homepage being redirected to 127.0.0.1
There are no IP addresses blocked now, or when I first reported the issue.
WP security > Firewall > Internet bots if below option is enabled please disable it. and cross check if it solves the issue
This option was not previously enabled and continues to be disabled.
Ban POST requests that have a blank user-agent and referer:
This part of your reply seems to have gotten truncated but “Ban POST requests” was not previously enabled and continues to be disabled.
There is no strict pattern to the problem; it does not happen at the same time of day, nor with the same increment between days, nor for the same amount of time while it is happening. We do see it in the wild roughly 2-6 times per month.
Our plan for the next time this is triggered will be to clear the site cache manually and see if this response is at least being cached. If so, the time after that we can disable the security plugin and then clear the cache to isolate this to AIOS.
More as we have it, but we are definitely open to more feedback in the interim.
Marking as resolved.
Thanks MBR, I did discover and enable this ‘Rename Login page’ option. Immediately after enabling it the email notifications of blocked IPs stopped arriving.
Since this new login page is not in the site index any attempts to scan and find it will likely trigger the server firewall.
Ultimately this is probably the best solution because it does not fill the DB with miles of logs. If I had continued down this path the next feature request would have been to auto-prune the blocked IP logs so the DB doesn’t get too big. XD
Cheers!
+1 for Curtis’ suggestion. Let’s get this baked into the next version!
Forum: Plugins
In reply to: [WP-FFPC] [Plugin: WP-FFPC] Backend status: down?Strange, I swore we had memcached and yet there it is: working using the “memcache” option. I now see:
Backend status: 127.0.0.1:11212 => up & runningCheers.
Hey kpgraham,
I don’t absolutely need, just providing some feedback.
When reviewing you might consider hooking into the rich text editor that WordPress proper uses. That might reduce the weight a bit as well as have these widgets look like the page/post editor which would be nice.
Also: Don’t forget that there are also a lot of Chrome and Safari users out there!
Cheers.