Big phpXXXXXX files in /home/USER/.cagefs/tmp

  • rodriguezzz

    (@rodriguezzz)


    6 years, 7 months ago

    Hi all.

    I have some users with huge tmp files (between 500MB and 2GB) in /home/$USER/.cagefs/tmp/phpXXXXXX. I noticed that if I configure high values for “max_execution_time” and “lsapi_backend_max_process_time” this tmp files appears to grow infinitely.

    Example of temp files (with “lsapi_backend_max_process_time = 300”):

    [root@server ~]# ls -larth /home/$USER/.cagefs/tmp/php*
    -rw——- 1 $USER $USER 700M ago 24 10:47 /home/$USER/.cagefs/tmp/php3bLX6P
    -rw——- 1 $USER $USER 752M ago 24 10:48 /home/$USER/.cagefs/tmp/phpyQVlXK
    -rw——- 1 $USER $USER 1,2G ago 29 05:00 /home/$USER/.cagefs/tmp/phpTVL8ud
    -rw——- 1 $USER $USER 1,2G ago 29 19:04 /home/$USER/.cagefs/tmp/phpEWRNjt
    -rw——- 1 $USER $USER 1,2G ago 29 21:16 /home/$USER/.cagefs/tmp/php5IRtuJ
    -rw——- 1 $USER $USER 1,1G ago 30 05:47 /home/$USER/.cagefs/tmp/phppCg2ag
    -rw——- 1 $USER $USER 1,2G ago 30 15:31 /home/$USER/.cagefs/tmp/phpFvflBt
    -rw——- 1 $USER $USER 1,2G ago 30 18:55 /home/$USER/.cagefs/tmp/phpl438Jl
    -rw——- 1 $USER $USER 1,2G sep 1 04:40 /home/$USER/.cagefs/tmp/phpnvjn9N
    -rw——- 1 $USER $USER 1,2G sep 1 05:06 /home/$USER/.cagefs/tmp/php5afzuV
    -rw——- 1 $USER $USER 1,2G sep 1 11:52 /home/$USER/.cagefs/tmp/phpO427u7
    -rw——- 1 $USER $USER 1,2G sep 1 14:39 /home/$USER/.cagefs/tmp/phpbRucIl
    -rw——- 1 $USER $USER 1,2G sep 3 05:57 /home/$USER/.cagefs/tmp/phpie8a2k
    -rw——- 1 $USER $USER 656M sep 3 16:23 /home/$USER/.cagefs/tmp/phpHNSknY
    -rw——- 1 $USER $USER 818M sep 3 16:24 /home/$USER/.cagefs/tmp/php5H7kSj
    -rw——- 1 $USER $USER 1,2G sep 3 17:41 /home/$USER/.cagefs/tmp/phpJiCASm

    I verified that if I disable wordfence this tmp files are not created.

    System info:

    Wordfence 6.3.18
    Wordpress 4.8.1
    CPanel 11.66.0.17
    CloudLinux release 7.4 (Georgy Grechko)

    I see this problem in several users in different servers (All with the same config as above).

    Any ideas?

    Regards.

Viewing 15 replies - 31 through 45 (of 57 total)
1 2 3 4
  • Plugin Author WFMattR

    (@wfmattr)

    6 years, 7 months ago

    Three questions for anyone who is seeing this issue:

    1. Has anyone found a way to reproduce the issue on demand? (Repeating this question since it might get missed due to the page break.)

    2. If you’re able to use strace capturing the issue in progress as Lucas did, can anyone send me a longer copy? The strace output helps, but seeing what happened further back would provide more detail. (You can email it to me at mattr at wordfence dot com.)

    3. Can you tell me what Apache MPM you are using? (e.g., prefork, worker, event..)

    -Matt R

    LucasRolff

    (@lucasrolff)

    6 years, 7 months ago

    @wfmattr:

    > 1. Has anyone found a way to reproduce the issue on demand? (Repeating this question since it might get missed due to the page break.)

    Not yet sadly :/

    > 2. If you’re able to use strace capturing the issue in progress as Lucas did, can anyone send me a longer copy?

    I’ll try to get a longer strace when I return from US

    > 3. Can you tell me what Apache MPM you are using? (e.g., prefork, worker, event..)

    We see the issue on both mpm event and worker – most of our infrastructure runs mpm event though.

    oga23

    (@oga23)

    6 years, 6 months ago

    Hi, guys,
    same problem here.
    TOday i got an email that my disk usage almost reached the quota. After an mail to my host where they told me that cagefs/tmp/phpxxxx (xxxx means something) file is growing fast.
    I don’t anything about this so i can’t really help but i am on php7, with wordfence activated. opcache isn’t activated.
    clearing caches(w3 total cache, wp fastest cache) didn’t help.
    hope someone finds a solutions real quick.
    I was on php5.6 and shared hosting and had same problem, that’s why i upgraded my hosting.
    Fingers crossed. Sorry i can’t help.

    Plugin Author WFMattR

    (@wfmattr)

    6 years, 6 months ago

    Hi all,

    I’m still looking into this issue, but I don’t have a site where I can reproduce the problem yet, so I still need to get some more details if possible. I’ve checked out PHP’s C source though, and the only place I can find where PHP is writing to the same temp file where the POST body should be (shown in the strace from Lucas) is when php://input is read, which is likely what triggers the issue, when the POST data is bad. In the strace and in the error message from Rodriguezz including “user_get_body(tmpstackbuf, 16384): read from client failed”, it looks like the connection between lsphp and the web server (or lscgi?) is broken when lsphp tries to read the POST body — PHP stops writing to that temp file when this occurs and truncates the temp file, but it will start writing to it again if php://input is read.

    As a test, if you have some sites where this issue is occurring regularly, try adding this to the wp-config.php, before the “stop editing” comment:
    define('WFWAF_DISABLE_RAW_BODY', true);

    If the site has Wordfence’s “Extended Protection” set up using auto_prepend_file, place that line in wordfence-waf.php in the site root, before the “if” statement, instead. (Don’t enter it in both files.)

    The change above is for troubleshooting — if this seems to make a difference, it will confirm if reading php://input is the cause, but isn’t intended to be a permanent fix. Let me know if it seems to help on any sites — if you need more detail about where to put that line, let me know.

    If there is any additional logging you can enable on lsphp, that may help narrow it down — it could be something to do with the lsphp child reaching the maximum number of requests, or some other limit, but I can’t tell from the available information.

    Setting a lower lsapi_backend_max_process_time could also help limit the file size, but that isn’t a permanent solution of course. Ideally, lsapi or PHP itself needs to prevent the runaway temp file writes when the POST body can’t be read properly, but we may be able to work around it for now, once the cause is confirmed.

    -Matt R

    Plugin Author WFMattR

    (@wfmattr)

    6 years, 5 months ago

    Hi @turistinfamilie,

    Are you able to try the test above, to see if this line of code helps? The location to place it depends on your setup and is described above:
    define('WFWAF_DISABLE_RAW_BODY', true);

    It may take a while for caches to expire, so it may not be immediate, but I’d expect it not to take too long. If it helps, it’s only a temporary fix — but confirming whether that works will help decide how we can address it. Since we haven’t reproduced the issue here, I don’t have a way to test whether reading the “raw body” is the cause, but it looks likely in the source code of PHP itself.

    -Matt R

    cnrt

    (@cnrt)

    6 years, 5 months ago

    I found this thread via a google search for the problem I am having ie tmp in .cagefs filling up with php4GejuR type of files and I wanted to know if I can delete them to save space. I run a website but am not a technical guy beyond a bit of php and html.

    But this only occurred after I changed my site to try and resize images to fit a certain sized slider on my site – code below. When I look at the phpblahblah files they all seem to mention adobe photoshop or a camera name or something re an image. So I think that GD when looking at the properties can’t cope with a big photo file and dumps this file. I don’t get it using wordpress, I’ve not heard of wordfence and I’m on the following
    cPanel Version 66.0 (build 33)
    Apache Version 2.2.27
    PHP Version 5.3.28
    MySQL Version 5.5.57-cll
    Architecture x86_64
    Operating System linux

    I am hoping this may be of help to you all in your deliberations – if not sorry for cluttering the forum.
    Nick

    My image resize code is as follows……

    $file_headers = @get_headers($geturl);
    if($file_headers[0] == ‘HTTP/1.1 404 Not Found’) {
    //$geturl = “img/noimage.png”;
    $url =”/img/noimage.png”;
    $pics=”0″;
    //$new_width=”183px”;
    $urlDescription=””;
    }else{

    $image_info = getimagesize($geturl);
    $width = $image_info[0];
    $height = $image_info[1];
    if ($height == “”){
    $height = ‘555’;
    $width = ‘833’;
    }
    unset ($imgage_info); //free up memory
    $thumbHeight = 555;
    if ($height < $thumbHeight){
    $new_width = floor( ($thumbHeight / $height)*$width);
    }else{
    $new_width = floor( $width / ($height/$thumbHeight));
    }
    // end of resizing
    }

    Tina Hammar

    (@tina-hammar)

    6 years, 5 months ago

    I had the same problem. I uninstalled wordfence and the problem was gone. Using another security plug-in instead. Sad.

    Anna Webdesign

    (@anna-webdesign)

    6 years, 3 months ago

    Goodmorning!

    Does anyone has a solution yet?
    I have the same problem 🙁

    I tried this:

    define('WFWAF_DISABLE_RAW_BODY', true);

    But that did not do the trick 🙁

    GeekyJules

    (@geekyjules)

    6 years, 3 months ago

    I too would like to know if this problem has been solved yet. I updated my server to Event MPM and Lsapi, and within 24 hours, Wordfence caused 80 GB of files in cagefs tmp. I pay for premium and had to turn off the firewall. On sites on my server where I have yet to update the firewall settings after the server upgrade, there hasn’t been an issue.

    GeekyJules

    (@geekyjules)

    6 years, 3 months ago

    A quick follow up: I turned off both real time traffic reporting and the firewall (including extended protection) and it’s still happening. Since my last comment, this issue has resulted in 50 GB of files being written to cagefs/tmp

    Plugin Author WFMattR

    (@wfmattr)

    6 years, 3 months ago

    @geekyjules: Since you have Wordfence premium, can you enter a support request at https://support.wordfence.com/ with the details above? We can provide more in-depth support that way, and the forum rules don’t allow us to provide support for the premium version here. Please also include a link to this forum post, and mention that “Matt R” has been looking at this issue.

    @anna-webdesign: Thanks for trying that line of code — if your site is using “Extended Protection” (shown on the Wordfence Firewall page), can you confirm that you entered it in wordfence-waf.php, after the opening “php” tag, but before the “if” statement? Or if your site is only using “Basic WordPress Protection” that it was entered in wp-config.php, before the comment that says “That’s all, stop editing”?

    @tina-hammar: I saw your reply a while back — but if you want to keep an eye on this post, I’m sure we’ll find the cause and work around it.

    Thanks for the few new replies — every bit of information we can get will help, since it’s an issue we haven’t reproduced on our own test servers. Based on the earliest information we had about this issue and sample files sent outside of the forum, it’s caused by PHP writing temporary data that isn’t even needed by Wordfence, though it’s triggered by something in our code.

    -Matt R

    Anna Webdesign

    (@anna-webdesign)

    6 years, 3 months ago

    Goodmorning Matt,

    Thanks for your response!

    My website is not using the extended protection.
    It is in de wp-config.php file before the comment:”That’s all…”

    I have WordFence on many websites – it is only happening to 2 websites.
    The websites are on the same hosting and both have woocommerce.

    Other websites on the same hosting witout woocommerce don’t have that problem. Maybe there is something in there?

    Anna Webdesign

    (@anna-webdesign)

    6 years, 3 months ago

    This weekend I deactived and deleted WordFence on one of the websites. The website where I deleted WordFence stopped getting big files.

    I really don’t want to stop with WordFence…Any idea how I can fix it?

    Anna Webdesign

    (@anna-webdesign)

    6 years, 3 months ago

    My hosting is telling me that it is caused by lsapi + PHP + Wordfence.

    It is a bug they can’t fix 🙁 because WordFence is triggering it.

    Maybe that can help?

    Plugin Author WFMattR

    (@wfmattr)

    6 years, 3 months ago

    We may have confirmed the issue — we’re going to watch a site for another couple days to be sure it’s no longer occurring, and then will begin testing a workaround. (Thanks @geekyjules for your help in testing this on your server!)

    @anna-webdesign: I agree with your host — I think it’s likely an lsapi/lsphp bug where the POST body gets lost (due to a broken pipe, based on the logs I’ve seen), but it might be possible for either Cloudlinux or the PHP developers to permanently prevent it from causing the giant temp files. They are created when that broken pipe happens, and Wordfence tries to read the POST body (which should just be a zero-length string if the body is missing). WooCommerce _might_ be involved as you mentioned — I see the read the POST body similarly, but I don’t think it’s on every POST request, just certain types.

    As a temporary fix, does your host allow you to choose any other options for how your site runs PHP, aside from lsapi? (fastcgi, cgi, suphp, or others may work, but may be a little bit slower depending on the host’s configuration.)

    -Matt R

Viewing 15 replies - 31 through 45 (of 57 total)
1 2 3 4
  • The topic ‘Big phpXXXXXX files in /home/USER/.cagefs/tmp’ is closed to new replies.