Support » Requests and Feedback » improve WP memory usage

  • I know WP has some issues they are addressing post 2.5. I want to bring up another big issue though that I don’t see getting much attention in the forums.

    I really think its time wordpress redoes its options table. I’ve discovered some things about memory usage and wordpress. When WP starts, one of the first things it does is load the options table into a GLOBAL variable. Maybe a long time ago this seemed like a good idea. However, this technique combined with how plugin authors have been told to use the options table has lead to a really bad WP memory issue.

    WP recommends update_option and get_option to plugin authors. Unfortunately, a lot of plugin writers haven’t thought about the implications of this. For example, the cforms plugin uses the options table to store every detail of form data. Everything from all the form vars, to email message template.

    Proof Test
    This test will work on any existing blog. For best proof tho, try this on a fresh wp 2.5 install. No plugins, no articles. Just a default setup.

    open index.php and add this line.
    echo "memory used before WP init: " . memory_get_usage() . "<br />";

    now. create a plugin. here is code. Once activated, this plugin will grab the memory being used by WP after init. I echo this in the footer of the page. For kicks, i’m printing $GLOBALS just so you can see how out of control the $GLOBALS var is.

    <?php
    /*
    Plugin Name: Mem Usage
    */
    
    function mem_init() {
      $GLOBALS["memused"] = "<br />memory usage after wordpress init: " . memory_get_usage() . " Bytes<br />";
    }
    
    function mem_foot() {
      echo $GLOBALS["memused"];
      echo "<br />memory usage after wordpress is done: " . memory_get_usage() . " Bytes<br />";
      print_r($GLOBALS);
    }
    
    add_action("init", "mem_init");
    add_action("wp_footer", "mem_foot");
    ?>

    Ok. now for some analysis. On my test setup, I get the following from viewing the blog front page.
    memory used before WP init: 11936
    memory usage after wordpress init: 6211472

    So before WP loads, php is using a mere 11Kb. After it is using 6.2MB. Why is this bad? An understanding of how apache and php work is necessary. Apache’s has its own memory footprint. It creates threads to answer http requests. Each thread stays alive for a certain amount of time or requests filled. (my server, an apache process stays open for around 1000 requests) the way apache gives memory to each thread is where we run into trouble. It will give a thread the most memory it needs. It allows this memory to grow as needed. However, it does not release this memory. It holds onto the memory for the life of the thread. This saves reallocating memory costs.

    So now for php’s play into this. say a fresh apache thread is 20M of memory. a php load that does nothing more than output text will use 11-12 KB of memory. so my fresh apache thread is now 20.12M with wordpress however, this shoots to 6.2M. so my apache thread is now 26-27M. Ok, this doesn’t break the bank on most sites. However, add a few plugins and add some traffic and you will find your site very quickly slow to a crawl as system memory is exhausted.

    Lets now test this by adding a popular plugin. Try adding cforms. don’t even bother setting it up. just activate it. Now view the site.

    On my test site, memory usage on the blog front page is now
    memory used before WP init: 11936
    memory usage after wordpress init: 6701800 Bytes

    Whoa. We just shot up by 500K. Pre WP 2.5, having several cforms would jump memory a significant amount. the cforms author has fixed that though in his latest version. His plugin is still using too much memory though and its because WP encouraged folks to use the options table. (or we could say WP isn’t teaching people to use the options responsibly.)

    Now on most systems, the default max memory php is allowed is 8M. A default WP install + cforms is already approaching this limit. And this is with no articles and only in the init stage. WP grabs more memory as it does its work looking for articles. Add a few plugins and its likely you will see memory usage shoot passed 10M.

    So we come to my recommendation. The big problem is that WP loads the entire options table on init into $GLOBALS. WP itself has misused the options table recently with the all those vars that start with rss_ (i think related to the dashboard. i’ve seen forum posts on this. hopefully they are fixing it) as of this writing, my options table is 489K in size. I think 450K or so of it is those rss_ lines. I think WP needs 2 options tables. A table for WP to use for its options that it needs to use for WP to work. Then a table for all extras. I think the get_option function for this extra options table should not store the option in a global. I think it should just do the sql lookup each time. I know this could be asking a little more out of mysql, but mysql caches these query types anyway. We are doing more harm storing all these options in memory than we would by letting mysql do its job. That is just my opinion. All I really care about is that the WP authors and the WP plugin community wisen up to the effects on memory we are having. There is a responsibility of WP to fix this issue. There is also a responsibility for better programming practices by WP plugin authors.

    A note. this message follows a lot of work I did recently on a site with server issues. On this site, I still see apache threads using upwards of 50M of memory. Most threads avg 35M of usage. His 50k users per day really makes this memory issue really obvious. I’ve got to imagine even wordpress.com would benefit greatly from addressing this issue.

    another big memory issue. this is prolly better saved for another topic. but in summary.. wordpress’s sql functions. The get_results method loads every single sql result into an array. This is awful. Why are we not looping thru using mysql_fetch per row. doing something with that row, then get the next row. WP would vastly improve performance by simply improving its usage of sql get functionality in the displaying of posts and comments.

Viewing 15 replies - 1 through 15 (of 20 total)
  • Thread Starter jbauguss

    (@jbauguss)

    well, for anyone that read this and cared. I’ve read elsewhere here in the forum that the powers that be don’t actually participate in the forum. So, I’ll prolly take this to the mailing list.

    I have seen in the mailing list that caching and memory is a big deal to them and they are looking to do a huge deal of work on it this summer.

    There are also apparently more advanced options in the use of get_option and update_option to prevent it from autoloading plugin data stored in options.

    I guess what really is lacking is good documentation.

    I also noticed that between my WP 2.3 and 2.5, with in average 30k unique users per day, the number of sql open queries has horribly increased, you may want to investigate it too :-/

    For instance, my “archives” plugin, asking from the database the list of all my 700 posts to this date, simply doesn’t work anymore with WP version 2.5. First I thought the plugin was incompatible and tested 4 other archives plugins, and every time a plugin tried to create of list of all blog notes, same as before, blank page without result.

    I’m not skilled enough to tell if WP 2.5 isn’t remembering to close what it has opened, or if it forced to open much more stuff than version 2.3, all I can do is witness the problem…

    Since it’s very relative to your problem, I shared the info 🙂

    yes the rss_ “options” are abosolutely killers of memories and cpus!
    they store entire posts of the rss feeds that are shown in the dashboard.
    If you limit those two boxes in the dashboard to only show 1 post each (you cannot really disable that….) it takes WAAAAAY less memory. In fact, those “rss_” options are not set to autoload and therefore they won’t affect other pages…. just the dashboard.

    Does wordpress also consume graphic memory?..Just kidding
    I think WP-Cache can help. See this

    i’m in a similar boat, i think. the last few days since I went to 2.5.1, the load on my VPS has made it unstable and basically unusable. the host says my MYSQL memory use has skyrocketed and are suggesting I need a dedicated server. With a MAX of 1,000 users a day (and most days it’s more like 150), I have a hard time imagining why this would be.

    I am having this problem currently. New install of the latest WP. I am backing up a website of mine, moving static pages to WP pages. I’m not sure when I hit my limit, as I loaded about 20 pages today. But yeah, I get a server error when I try to go to edit-pages.php in the admin. Seems like it won’t load them, which totally messes up my navigation and the rest of the site won’t load properly now, either. I think total, I have 73 pages, and was planning on having something closer to 150-170. This isn’t even blog posts, just pages. So – not sure. I might jsut have to rethink my plan to move my site to WP. It doesn’t seem to be able to handle it.

    Same problem here. If anyone is interested in doing some consulting work to help me fix this (particularly, jbauguss), please feel free to contact me.

    Good luck to all!

    I’m having serious memory issues myself at the moment. I’ve tried both wp-cache and wp-super-cache and both seem to make the problem worse, rather than better.

    My site was shut down by my host for memory usage and moved to a temp server. I then moved my site to a VPS with 2gb of ram in the box, and my site was crashing the VPS daily.

    I don’t have what I would call an overly popular site. (About 700 RSS subscribers, according to Feedburner.) This is something that needs to be addressed.

    Okay. Just to update everyone on what I did with my site. I changed about 85% of my pages to posts using a page/post converter plugin and that cleared up all of my problems. For some reasons, more then 20 pages starts to make the system act real funky. But, I now have about 300 posts, and works very well.

    I‘m having a similar problem, but don’t have many pages so djeloso’s fix won’t solve the issue. Has anyone come across another solution?

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    On the original topic: WordPress pre-loads all the options that have autoload=true because those options are considered to be the options needed to actually build the page. It’s quicker to load all those options at once with one query and then cache the information than to load it on demand with dozens or hundreds of database queries. DB Queries are expensive, loading data one time is far less expensive.

    So, if your plugin needs the data to actually build the page, then having it autoload that data is the right thing to do. Since this is usually the way it works, it’s the default. If your plugin does not happen to need this information every page load, then the option functions allow you to set autoload to false manually, so your data won’t be loaded automatically.

    The RSS options are a good example of this. If you’re displaying RSS feeds on your page, then they need to be loaded into memory in order to be displayed. This cannot be worked around, PHP has to load the things before it uses them. Making it load them separately is silly. This is why they are autoload, because they have to be loaded anyway, might as well get it done as part of the single mass load.

    This is not a bug or a design flaw. It’s intentional, and it’s faster than doing it some other way.

    The get_results method loads every single sql result into an array. This is awful. Why are we not looping thru using mysql_fetch per row.

    Because loading all the data at once is faster. Really. That data has to be processed and such, and while you’re sitting there holding a database connection open with result data and processing that data one row at a time, another thread could be using those resources that you’re taking.

    You have to consider impact from both sides. You want to use that database connection to the max and then stop, not use it for a while, wait, hit it again, wait, hit it some more, etc. Get your data and get the heck out of there. Then process on your own time.

    Otto42 – Do you have any ideas what could be causing WP to use to much memory?

    As I wrote about here and here, my blog is using 660Mb of memory when my account is a 256M VPS. As a result my hosting company in requesting that I upgrade to the next hosting package ($90/month), which I can’t afford. My only other option is to reduce how much memory WP uses, but I have no idea how to do that.

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    The general answer to this is WP-Super-Cache. It reduces memory and CPU use to very low numbers, because it is a whole-page-cache. No memory is used when your site serves up a super-cached page, as it simply reads it from disk. No executable code, no database hits, nothing.

    That said, if your site is exceeding their limits, then it’s either time to up the limits or switch to a new host that offers a better deal. A site will take up memory and cpu usage in direct proportion to the popularity of the site. You can’t get something for nothing.

    The problem I have with WP-Super-Cache is that the dynamic content in my sidebar won’t load anymore (for instance, Twitter Tools, Recent Comments, Random Quotes) and there are all aspects of my site that I like very much. 🙁

    And true, upgrading my package would certainly solve the problem but $90/month for hosting a foodblog seems extreme! Is there a list of more WP friendly hosts out there?

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    The problem I have with WP-Super-Cache is that the dynamic content in my sidebar won’t load anymore (for instance, Twitter Tools, Recent Comments, Random Quotes) and there are all aspects of my site that I like very much. 🙁

    There’s other ways to do those sort of things. Look for methods that use javascript to display your Tweets or Random Quotes instead. I know plugins like this exist, in both cases.

    The Recent Comments could be a minor issue, but generally wp-super-cache is smart enough to rebuild when new comments and such are added, so those cases where actual content is added to the site is not usually a problem.

    I have no hosting recommendations for you. I’ve heard good things about A Small Orange, but I have not used them myself.

Viewing 15 replies - 1 through 15 (of 20 total)
  • The topic ‘improve WP memory usage’ is closed to new replies.