WordPress 4.4 XML-RPC Exploits Still NOT Fixed

  • own3mall

    (@own3mall)


    8 years, 4 months ago

    Hi Guys,

    My server has been under attack quite a bit recently. After spending a lot of time investigating why the php5-fpm process was using so much CPU, I determined that several attackers have been targeting an installation of WordPress on my server.

    More specifically, the logs show attacks against xmlrpc.php. Using this information, I searched the net for answers. It appears this problem is known and has been known for years now.

    http://www.blogtips.org/block-wordpress-brute-force-attacks-via-xmlrpc-php/
    https://wordpress.org/support/topic/xmlrpcphp-attack-on-wordpress-38?replies=28

    Thinking that this problem may have been fixed in the latest version of WordPress, I updated my blog from version 3.6 to 4.4. I then unbanned the IP address that was attacking the WordPress installation, and sure enough, nothing has changed. The attack immediately takes my server load to over 10, and the CPU is hammered by php5-fpm stemming from the POST requests against xmlrpc.php

    Sample log:

    93.174.93.61 – – [16/Dec/2015:10:44:34 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:35 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:35 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:36 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:36 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:36 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:37 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:37 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”
    93.174.93.61 – – [16/Dec/2015:10:44:38 -0700] “POST /xmlrpc.php HTTP/1.0” 499 0 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)”

    Is this issue going to be fixed anytime soon? An exploit that can bring down an entire web server needs to be fixed. I cannot believe that it hasn’t already been fixed. I’m not sure how the attack works, but the request shouldn’t even be processed if it’s too large or contains too many login attempts.

    I believe software I’ve helped write before also uses XML-RPC, but the way we implemented that is that we don’t process a request unless a unique key has been sent to us that we expect.

    If you need more information from me or my webserver, please let me know.

    This problem needs to be fixed. I cannot have WordPress installations running on my server that overload my resources. I don’t believe the solution of blocking access to xmlrpc.php makes sense. It should be fixed so that this exploit is rendered useless.

Viewing 15 replies - 1 through 15 (of 25 total)
1 2
  • Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    8 years, 4 months ago

    The attack immediately takes my server load to over 10, and the CPU is hammered by php5-fpm stemming from the POST requests against xmlrpc.php

    That’s because you have not actually fixed the attack. The problem is not in WordPress, and nothing that WordPress does is going to prevent your site from being hammered. Your site would continue to be hammered even if they were hitting wp-login.php or index.php or even just the homepage.

    If you throw 5-10 requests per second at a server that cannot take the load, then it goes down. That’s your underlying problem here. Nothing in the PHP code will fix that.

    You need to either block the requests upstream, or setup something like fail2ban. WordPress can not fix what is the equivalent of simply getting too many requests to a site and overloading it.

    the way we implemented that is that we don’t process a request unless a unique key has been sent to us that we expect.

    That is the way WordPress works too. They’re sending usernames and passwords over and over and over until they get in. If you have a strong password, then they won’t get in, but that doesn’t prevent them from continuously beating on the door.

    Thread Starter own3mall

    (@own3mall)

    8 years, 4 months ago

    I’m sorry, but that’s not correct. I’m already rate limiting at 10 requests per second with a burst of 5 using nginx for all php files. My server is extremely powerful, but this attack is ridiculous. The IPs “hammering” my servers are sending less than 4 requests per second, which is nothing for nginx. However, their payload is making php5-fpm work like crazy. There is certainly something wrong here with WordPress. My server is about as tightly locked down as you can get. Nothing else is able to flood my server except for the xmlrpc.php exploit. They are sending multiple commands to execute in multiple methods in one request that the server is having to process.

    Surely there is a better way to lock this down? It’s only a few IPs from Ecatel LTD that seem to be exploiting this to the point that php5-fpm spawns several processes that use over 10% of the CPU. I wish I could specifically tell you what they are doing, but I’m not sure yet. I just know that this is not typical and something is being exploited regarding xmlrpc.

    The attacks come from a single IP address. I can ban these IP addresses which fixes the problem, but they are somehow exploiting WordPress code to DDOS the server in a non-DDOS fashion. They are not sending that many requests per second. It’s hard to stop these attacks.

    I already am using Fail2ban, but I fail to see why I need to use a third party utility to fix code that can be exploited in WordPress.

    nginx is capable of handling thousands of requests. There’s something about these POST xmlrpc.php attacks that is causing the server to execute a ton of code and work hard. That’s where the problem lies. It shouldn’t allow a huge payload or process a request that will trigger a bunch of actions.

    Many others have reported this problem, yet no action has been taken by WordPress developers. You cannot blame “too many requests” as the problem when that is not the case (not for me anyways, I can tell you that much for sure). There are plugins that disable access to the xmlrpc.php file. Clearly, something is exploiting this functionality, and people are fixing it by disabling access. I will do that globally too on my server if I have to, but I really find that quite sad that apps using this functionality won’t be able to work.

    When xmlrpc.php isn’t being exploited on my server, my average system load is 0.01, 0.10, 0.09. When it’s being exploited, it goes over 5 and sometimes 10. And I’m hosting over 100 websites on this server… that says a lot.

    I will try to capture packets the next time this happens. This needs to be stopped. I’m assuming this command should give me what I need:

    tcpdump -i eth0 src "EXPLOITING_IP_ADDR" -A -s 0 -w exploit.cap

    In my case:

    tcpdump -i eth0 src 93.174.93.61 -A -s 0 -w exploit.cap

    Then maybe we could find out what they’re doing?

    Thread Starter own3mall

    (@own3mall)

    8 years, 4 months ago

    It is premature to mark this topic as resolved. I’m not convinced at all that it is a server configuration problem especially when others have the exact same problem and people are writing plugins to disable the xml-rpc functionality completely.

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    8 years, 4 months ago

    However, their payload is making php5-fpm work like crazy.

    Their payload consists of attempting to authenticate. Nothing more than that, really.

    However, this means that WordPress itself and all the supporting code has to load, the database must be queried to retrieve the hashed password, and the password hash must be checked. By design, password hashes are slow and computationally intensive, to make them harder to crack. The same basic things happen when they were previously hitting the wp-login.php entry point.

    The main XML-RPC change in 4.4 was to eliminate the multi-call methods, so that they could not send 1000 password checks all at once.

    The upshot of it is that PHP is a bit slow sometimes. It’s a scripting language. One general speedup you could try is to implement good PHP code caching, such as OPCache or APC. But the general problem of “jerks hitting your site rapidly” is not something you fix at the layer of the PHP script code. If a request makes it to the PHP code, when it should have been blocked, then it’s already too late. It’s passed through your whole stack and taken longer than it should have. Your processes have spawned, and run, and caused additional load.

    Yes, WordPress could stand to have some optimizations. But no amount of optimizing will fix the general issue of a DOS attack.

    Thread Starter own3mall

    (@own3mall)

    8 years, 4 months ago

    OK, I captured some packets from the attacking IP address.

    They can be downloaded here:

    http://dinofly.com/files/captures.tar.gz

    It appears that pingbacks are being used in a Distributed Denial of Service attack somehow against my server or someone else.

    exploit_2.cap was a tcpdump lasting only a minute. exploit_3.cap was a tcpdump lasting only 10 seconds. The first was only a few seconds.

    Anyone know what’s going on here? I disabled pingbacks a long time ago, but some of my initial posts weren’t adjusted… time to run MySQL queries, but still, pingback functionality should be improved to stop these DDOS attacks.

    Seeing a lot of the following in the captures:


    [ ASCII binary gobbley gook redacted ]

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    8 years, 4 months ago

    Pingback spam. Yes, disabling pingbacks on your posts would help with that. If the pings for the post are closed, then no processing of the ping occurs beyond that point.

    You can use the Bulk Editor screen in WordPress to easily disable pings for any set of posts.

    Alternatively, if you don’t want pingbacks at all, this method works too, by simply removing the pingback.ping endpoint:

    https://wordpress.org/plugins/disable-xml-rpc-pingback/

    That’s not a DDOS, BTW. That’s just spam, trying to get links to some crappy escort site to show on your blog.

    Digico Paris

    (@digico-paris)

    8 years, 4 months ago

    Hi,

    Interesting discussion really.

    @own3mall yeah you should disable pingbacks at each single post level first. You could also try Cloudflare free plan to add a layer against that kind of hacks.

    I could also suggest you to use a list of authorized servers to access xml-rpc.php manually, through .htaccess /httpd.conf or nginx equivalent config files (eg, a rule like: allow from Google bot and instagram verified domains, deny from all).

    Thread Starter own3mall

    (@own3mall)

    8 years, 4 months ago

    I’m not sure how that isn’t a DDOS. It makes my php5-fpm spawn a bunch of processes that use too much of my CPU.

    The server ends up working too hard on pingbacks (for reasons I simply cannot understand), which is a DOS for sure.

    The posts they’re trying to pingback to don’t even exist. So, what’s making php5-fpm work so hard? Are the database queries simply that inefficient? They’re not even attacking posts that contain pingback support. In fact, the URL for the pingback doesn’t even work!!!!

    Example: http://blog.eamster.tk/ubuntu-grub-fails-to-install-on-raid-array/ is what one of the strings is set to in the packet capture. Yet, this page does not exist. The post is now: http://blog.eamster.tk/?p=372

    I had to disable permalinks on nginx, so all of my posts updated to new links since nginx doesn’t support .htaccess files like Apache2 does. Plus, after researching permalinks, I don’t think they’re a good idea… but that’s another story.

    Digico Paris

    (@digico-paris)

    8 years, 4 months ago

    @own3mall

    Well php5-fpm use caches and works like a background task, so if well setup and on a small VPS for instance, it can use in general up to 10%-20% of a very small cpu on a permanent basis, because it rewrites caches all the time (no DDOS, no bug there).

    After, yeah you’re right this workload on pingback is not normal at all but if secured, with advices from above discussion, or simply having them fully disabled, it should really improve situation.

    NB: Tried both urls provided and it’s the normal way of working, nginx says error and that’s the end of php use (less than 1ms to call nginx error page).

    Thread Starter own3mall

    (@own3mall)

    8 years, 4 months ago

    Yeah, I think I will install that plugin that disables pingbacks, but still, I think something is wrong here.

    The posts referenced by the attacker no longer load, the posts don’t support pingbacks already, and pingbacks in general is a feature that is easily abused which doesn’t serve much purpose other than causing DOS problems it would seem.

    nginx runs all of my other websites just fine. Like I said, the load is usually always under 0.30 unless someone is spamming or DDOSing my WordPress.

    I wonder if the web server change from apache2 to nginx and then updating the permalinking settings caused some sort of bug that is making pingback requests slow my server to a crawl for some reason.

    I would assume its safe to say that this is a WordPress problem considering the high traffic on my other websites which do not run WordPress.

    Digico Paris

    (@digico-paris)

    8 years, 4 months ago

    @own3mall

    You don’t need to use a plugin to disable pingback and trackbacks, it’s in Dashboard > Settings for whole WP site. Sometimes you have to force it in the database at each post level (especially with migration, just make a big MySQL request to disable it for all posts, it’s easy really).

    Alright if general load is 0.30.

    Just discussing, yeah this move created for sure issues with your permalinks, as Nginx doesn’t work like Apache. My guess is, you have to dig in Nginx config files and MySQL wp_options to find potential issues or misconfiguration issues.

    I’ve seen this case in Apache to Nginx where posts where auto pinging them, (auto ping back), while being moved. And it created a lot of fluff.

    WordPress or not, just trying to help or give potential ideas to solve the issue,

    Good luck,

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    8 years, 4 months ago

    Even if the post no longer exists, WordPress has to load and check the database. That’s already a lot of load to account for.

    Digico Paris

    (@digico-paris)

    8 years, 4 months ago

    @samuel

    Thanks for the feedback, I wasn’t aware of that. And probably own3mall problem if WP loads even for unaccounted posts id for each permalink.. and in server migration.

    Thank you,

    Thread Starter own3mall

    (@own3mall)

    8 years, 4 months ago

    Thanks, I installed the disable-xml-rpc-pingback mod and hope that will solve the problem. I will update this thread if it doesn’t.

    https://wordpress.org/plugins/disable-xml-rpc-pingback/

    WordPress should disable pingback functionality by default. It’s too easily abused and is evidently not very efficient. You shouldn’t blame sever operators for this issue. WordPress provides the functionality, WordPress should fix and secure it. As it stands right now, pingbacks introduced another way to DOS the internet. It affected me when my server was running Apache2 and nginx. Every webserver running WordPress is essentially open to this issue.

    SGURYGF2

    (@sgurygf2)

    8 years, 4 months ago

    I too believe that this is something that could have been addressed by WP at some extend.

    One way to mitigate this issue is to count how many failed authentication requests are sent and if too many, then ignore the requests. No legitimate connection will initiate that many requests per second. It doesnt have to be on failed authentication requests but on any request.

    Of course this will only work on minor DoS attacks not big ones but thats still something.

    I think that if WP team wanted to deal with this, they could of.

    I also believe that it CAN be considered as a vulnerability because its a vulnerable point that can lead to DoS attacks.

    One could argue that this can and must be handled via 3rd party software like fail2ban or a firewall. I agree, but at the application level, im sure there are also things that can be done.

    Its better to make make WP 1% slower and more “secure” rather than having xml-rpc attacks slow the website down by 100%.

Viewing 15 replies - 1 through 15 (of 25 total)
1 2
  • The topic ‘WordPress 4.4 XML-RPC Exploits Still NOT Fixed’ is closed to new replies.