“500” errors, server crash and redirect errors

  • wpfranner

    (@wpfranner)


    12 years, 3 months ago

    Hello all

    On my WP multisite server with 200+ sites in subfolders, I am experiencing a lot of different errors.

    Problem overview
    1) “500” error (possibly linked to the “PHP Fatal error: Allowed memory size of 268435456 bytes exhausted” that I see in my apache log)
    2) Server crashes every 3-4 days.
    3) Getting several thousand “Request exceeded the limit of 10 internal redirects due to probable configuration error” errors a day in my apache log.

    Problem description
    1) To try and fix this issue I have tried allocating as much as 1GB! of RAM in the PHP.ini file and in the wp-config.php file but to no avail. The server is bran new with 16GB RAM and 2 Zeon processors and is only handling this one site.
    The error occurs in network admin when trying to add users to a site or viewing installed themes. Other places will generate the error as well but not consistently. I have tried removing all plugins but “WordPress MU Domain Mapping” plugin which I can’t do without but still get the errors.

    2) The server crashes every 3-4 days with no real consistency. According to my server monitor system, it uses all available RAM and SWAP memory and stops reacting to requests. I have been going through the log files and nothing unusual is happening just before a crash. I have tried restarting the apache a few times a day which seems to reduce the frequency of the crashes. I also tried moving to another server but the problem remains.

    3) The “Request exceeded the limit of 10 internal redirects due to probable configuration error” errors are filling my log at an enormous rate. This problem is well described on wordpress.org and other places but I can’t find a solution.

    After having spent more than 2 weeks on this I am really hoping you can help me shed some light on the real reasons for the issues.

Viewing 15 replies - 1 through 15 (of 17 total)
1 2
  • Moderator Jan Dembowski

    (@jdembowski)

    Forum Moderator and Brute Squad

    12 years, 3 months ago

    1) “500” error (possibly linked to the “PHP Fatal error: Allowed memory size of 268435456 bytes exhausted” that I see in my apache log)
    2) Server crashes every 3-4 days.
    3) Getting several thousand “Request exceeded the limit of 10 internal redirects due to probable configuration error” errors a day in my apache log.

    Well THERE’S your problem. Except for item #3, you’ve got memory issues. Item #3 means your site is successful, or you’re being DoS’ed. Either one will do that.

    To try and fix this issue I have tried allocating as much as 1GB! of RAM in the PHP.ini file and in the wp-config.php file but to no avail.

    I also tried moving to another server but the problem remains.

    It is almost certainly a software issue and an amok plugin or theme will do that. One plugin did that exact same thing for my install and I get 1/100 of your traffic. Adding more RAM won’t fix it because with enough requests you’ll run out of any size pool.

    Can you get your host to graph out memory utilization? On my server I was able to install Munin and once I found the errant plugin I was able to confirm that I kept below the physical RAM on my VPS.

    You’ll need to get medieval with your plugins until you find the problem. Since you’ve hundreds of sites that may not be practical so your need to profile memory usage of your themes and plugins.

    Andrea Rennick

    (@andrea_r)

    12 years, 3 months ago

    Do a top via ssh. Look at your access logs – are you being hit by bots repeatedly?

    Thread Starter wpfranner

    (@wpfranner)

    12 years, 3 months ago

    @ Jan Dembowski
    I have contacted my host and expect an answer regarding Monin soon.
    After your replay I have setup a ghost server to test whether a plugin is causing this problem or not. I will finish that test later today.

    @ Andrea_r
    I have talked to my host and they say that neither there firewall or network monitor system has detected any irregularities that would suggest an attack.

    Any other suggestions as to what plugin is the most likely to be the instigator?
    Plugin list:
    Exclude Pages from Navigation
    Hierarchical Pages
    MapPress Easy Google Maps
    MCE Table Buttons
    Members
    No Comments On Pages
    Spectacu.la Page Widget
    WordPress MU Domain Mapping

    Moderator Jan Dembowski

    (@jdembowski)

    Forum Moderator and Brute Squad

    12 years, 3 months ago

    Having Munin or similar will at least graph your memory usage so you can identify around when you’re about to hit the memory ceiling. And it’s not necessarily an attack as much as a flood of requests from search engines. Identifying the trigger helps find the issue.

    That’s not a large number of plugins and identifying if it’s one of those is a trial an error exercise. When I ran into this, it was a Flickr plugin for me.

    What about the themes you have installed for your sites?

    Andrea Rennick

    (@andrea_r)

    12 years, 3 months ago

    What Jan said. 🙂

    We have a long term site which is pretty quiet now, yet needs to be on a dedicated server just to handle things like bots and google coming around looking for feeds that haven’t been updated in years.

    Thread Starter wpfranner

    (@wpfranner)

    12 years, 3 months ago

    @ Jan Dembowski
    We only have one main theme with about 30-40 childthems on the server. They are all made with the “Thematic” template.

    @ Andrea_r
    I can’t see any evidence of bot activity in the logs. Do you know where I should be looking and what for?

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    🏳️‍🌈 Advisor and Activist

    12 years, 3 months ago

    Look for a bunch of gets, all in a row, like this:

    /wp-content/themes/newsworld-1.0.0/thumb.php?src=/g0../0d1.
/wp-content/themes/DeepFocus/tools/timthumb.php?src=/g0../0
/wp-content/themes/TheProfessional/tools/timthumb.php?src=/
    Moderator Jan Dembowski

    (@jdembowski)

    Forum Moderator and Brute Squad

    12 years, 3 months ago

    Ha! I totally forgot about that pseudo DoS. Until I did the .htaccess thing, I kept getting smashed too.

    wpfranner if you do see those thumb.php GETs in your log, give this a read.

    http://tech.ipstenu.org/2011/timthumb-pseudo-ddos-effect/

    Andrea Rennick

    (@andrea_r)

    12 years, 3 months ago

    Do you know where I should be looking and what for?

    anything that has the word “google” in it. 😉 I’m not being facetious either – I’ve had to temporarily block it and force them to come by less often.

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    🏳️‍🌈 Advisor and Activist

    12 years, 3 months ago

    +1 Andrea :/ Bing Bonged me the other day.

    Thread Starter wpfranner

    (@wpfranner)

    12 years, 3 months ago

    I looked around for a plugin or script that could curse the memory error. I found something really strange. Somehow, the users on certain sites have started replicating almost endlessly. On one site I found that one of the Admin users was duplicated 25.000 times… see this image

    The user only appears once on the list but in the _usermeta table in DB it is duplicated A LOT (90.176 lines of data). There are several of our users that this has happened to according to the DB table.
    The usermeta table is now at 16MB in raw sql dump…. that doesn’t really seem right.

    Does anyone have an idea on what is happening here?
    We have the Members Version 0.1.1 By Justin Tadlock installed on the server. Could that plugin cause this?

    Andrea Rennick

    (@andrea_r)

    12 years, 3 months ago

    I would deactivate it and other user plugins to rule them out at the very least.

    (good hunting)

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    🏳️‍🌈 Advisor and Activist

    12 years, 3 months ago

    I have Members on a couple times without any issues, FWIW. But yeah, that looks way shady to me!

    Thread Starter wpfranner

    (@wpfranner)

    12 years, 2 months ago

    Solution
    So I finally got the answer to the problems I described. It was indeed the _usermeta table that was causing the problems.
    After emptying the entire _usermeta table, the serverload went down by almost 70 %! I had to remap all the users to the different sites which took almost 12 hours.

    Now my sites are lightning fast and I get no errors at all.
    As to what caused the _usermeta table to run amok in the first place is anyone’s guess. I have updatet all my plugins and WP and no user copying has taken place yet.
    Thanks for the help all!

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    🏳️‍🌈 Advisor and Activist

    12 years, 2 months ago

    What plugins were/are you running?

Viewing 15 replies - 1 through 15 (of 17 total)
1 2
  • The topic ‘“500” errors, server crash and redirect errors’ is closed to new replies.