[resolved] high traffic crashing usmagazine.com (52 posts)

  1. WillHines
    Posted 10 years ago #

    Short version: Looking for performance tips for a high-traffic, high-profile site.

    More details: I'm a web developer working on a web site for usmagazine.com (site of Us Weekly) that's running on WordPress, based on my recommendation. The staff loved the interface, but high traffic is overwhelming the site. Database errors. There's other solutions -- more powerful server -- but WordPress seems to take the heat when I call tech support. I'm trying to save it.

    Any ideas?

    * I've installed Wp-cache, but it's not enough. (Is Staticize a better bet?)
    * Looking into disabling comment counting.
    * There's no category list or archive list in the sidebar.
    * This is a dedicated virtual server.
    * Traffic ranges from 36,000 to 72,000 sessions a day.
    * MySQL 4.x.
    * Other plugins: Spam Karma, Paged Comment Editing, fl_movie.

    I understand that the problem is high-traffic is beyond optimizing queries and installing plugins. But I'm asking for things like that because those are things I can do in the next few days.

    Lots of good ideas in these forums. An amazing community! WordPress rules. Thanks for any advice.


    WordPress database error: [Can't create/write to file '/tmp/#sql_16d8_0.MYD' (Errcode: 17)]
    SELECT DISTINCT * FROM wp_posts WHERE 1=1 AND post_date_gmt <= '2006-04-26 04:29:59' AND (post_status = "publish" OR post_author = 2 AND post_status != 'draft' AND post_status != 'static') AND post_status != "attachment" GROUP BY wp_posts.ID ORDER BY post_date DESC LIMIT 0, 10

    Odd side effect: I was mailed a backup of the database about 45 times before I disabled the backup and wp-cron plugin.

  2. WillHines
    Posted 10 years ago #

    MySql 4.1.18

    Still checking through the forums. I wonder why I typed that. But I did.

  3. WillHines
    Posted 10 years ago #

    I understand now that it's not really the kind of thing that can be helped on a support forum. But I'm going to document this problem in case it's helpful.

    Our ISP is mediatemple.net, and I've been very happy with their support up till now. I think Zeldman gives them a shout-out in his blog. However, them quickly saying "It's WordPress" doesn't cut it for me. Even if they end up being "right", it smacks too much of "let's point at the one component we didn't install." I should point out they still respond very quickly and are polite, etc.

    We've had indications of traffic problems before this and have asked about bumping up from a dedicated virtual server to a dedicated physical server. They seem oddly not interested in talking about that.

    I'm not rebuilding the site just because my ISP won't help examine where the bottleneck really is. Even if I abandon WordPress, wouldn't ANY php-mysql app hit similar limitations with huge traffic?

    Last night, I was getting error code I posted above. I was going to rebuild the database, courtesy of the daily backup I get from Scott Merrill's plugin. But if there were disk errors, I was worried about somehow aggravating the problem by deleting and writing a bunch of data.

    Stuff I've done/looked into:

    * My ISP has asked if I can have a flat HTML front page. I've installed WP-cache (which my tech support guy found for me). Not truly a flat page, but I assume this is helping.

    * After looking through these forums, I installed Jerome's Query Diagnostics to see if any queries were holding us up.
    This was for the front page.
    This was for a single post.
    Nothing stood out.

    * It seems comments can cause problems, but comments are a big part of the site and I can't disable them, or not for long, anyway.

    * I cleared out the default entry (pingomatic?) in the "update services" box.

    * I disabled trackbacks and pings (we never had many anyway).

    * We don't have a sidebar, so it's not like there's a list of categories or past entries whose queries could be slowing us down.

    * We have plenty of disk space left -- 10 gig of 17 available.


    This morning, I get this message from mediatemple:

    "Due to a previously undocumented problem with the virtualization software, some users have been experiencing situations where they are exceeding their kmemsize limits."

    Not sure what a "kmemsize limit" is, but I assume it is usmagazine's traffic that has created this problem. I'm hoping they can make hardware adjustments to allow me to keep using WordPress.

    Let me say again that it's amazing how well-documented the support for WordPress is, and amazing how easily I'm able to add plugins, and locate information on problems. I love it.

  4. manstraw
    Posted 10 years ago #

    In the very least, get yourself a dedicated server. Get the fastest one you can, with tons of ram. A site like that needs it. And make sure it's in a fast data centre with big pipes to the internet backbone. And don't put email on the same server. You might be alright with that.

    If you can spend the money, considering something like this > http://ev1servers.net/Dedicated/RTG/servers/Monster.aspx. Please note I'm not recommending ev1 necessarily, please shop around, I just provided a link for information on what a high end server might feature. If the budget can't afford that, get the best with the money available.

  5. [Can't create/write to file '/tmp/#sql_16d8_0.MYD' (Errcode: 17)]

    This suggests that it's possible the temporary directory on the server is filling up which you should contact your host about.

  6. WillHines
    Posted 10 years ago #

    Thank you for the advice.

    I have asked repeatedly for prices of a dedicated server. They're telling me that the software (WordPress) is so inefficient that IT is the bottleneck, not the server. That's not their words -- but they shrug off upgrading as a good solution, pointing to "WordPress." It's frustrating.

    It seems like it's subjective to blame either the software or the hardware. But I guess I have to either do what my ISP says or switch ISPs.

    I'll search for an ISP that will stand by WordPress, but I imgaine I'll have to find a new blogging solution that they're willing to work with.

  7. angsuman
    Posted 10 years ago #

    I have recently faced a slowing down / hanging issue with WordPress wherein my Apache server was consuming massive amount of CPU. I too am on a Virtual Private Server and I use wp-cache 2 (recommended).
    On analysing the SQL queries the problem was isolated and fixed. Today my site runs like a charm.
    You can check out the article (linked above) for details.
    Few other thoughts:
    wp-cron is basically buggy (resulting in server overload). I have tested it in the past couple of days and have fixed it for my use. I haven't got around to releasing my version yet or inform the author. You may want to switch over to using a cron job instead as you are on a VPS.

  8. WillHines
    Posted 10 years ago #

    Cool. I'll check these out.

  9. davidchait
    Posted 10 years ago #

    What virtual server plan are you on? Have you paid a tech to adjust your PHP and MySQL settings to tune to the memory limits of your VPS?

    WP-Cache should be making the site almost no-query on cached pages. It'll hit apache, light php, and otherwise return quickly.

    I'd agree with westi - sounds like there's something wrong with the server configuration overall. I'm shocked that mediatemple isn't handling this better (they're rated highly by many users...), and the response that WP is 'so inefficient' and that moving to a dedi wouldn't help is just irresponsible.

    If you're going to host, find a host that actually supports/manages their VPS plans heavily. You need a managed service, or a tech who deals with this stuff daily/weekly and can lend a hand (maybe at a reduced cost..). There's a least one or two folks on the forums here that have dealt with VPS optimization -- but you might be beyond what a VPS can handle.

    Other thoughts:

    serving up lots of big, high res graphics and static content? run something like lighttpd/et al to serve up the static stuff. In some cases (someone who knows what they are doing) you could completely replace apache. There are also commericial/paid servers that have even better performance -- if apache performance is impacting things at all.

    wp-cache2. make sure it is on, and working properly.

    other plugins. be careful what you use, try to ensure they aren't adding heavily to the pageload time (either in code or in queries).

    your second query dump showed:

    [0] => SELECT option_value FROM wp_options WHERE option_name = 'siteurl'
    [1] => 0.23864006996155
    [0] => SELECT * FROM wp_users WHERE user_login = 'Will'
    [1] => 0.53467917442322

    Obviously, that's showing an overloaded server.


  10. angsuman
    Posted 10 years ago #

    Oh and you may try using Bad Behavior and / or Akismet instead of Spam Karma.

    > * I cleared out the default entry (pingomatic?) in the "update services" box.

    This will not help much as it affects only during posting time.

    > * I disabled trackbacks and pings (we never had many anyway).

    This may moderately help but I would think the real problem is elsewhere.

    I looked at your queries. By itself they don't look much of an issue. However the cumulative effect may be killing you.
    Use wp-cache 2 (linked above) and set cache time to at least 2 hours or more.
    Also try to get query details from SQL server itself instead of any plugins.

    > I'll search for an ISP that will stand by WordPress, but I imgaine I'll have to find a new blogging solution that they're willing to work with.

    My ISP supports WordPress, even installs it for you. They worked with me on my server load issue and actually helped me to identify the errant query.

  11. WillHines
    Posted 10 years ago #

    Wow! This is awesome. Thank you so much.

    Mediatemple has been nothing but great in many instances before. This is an anamoly. It's like I pissed them off.

    I'm working getting someone to help me isolate queries on the MySQL server.

    I have a feeling it's the comments. Our traffic is being driven up by an "exclusive" post on Britney Spears being pregnant. Yes, this is a hilarious problem. Viewers are commenting like crazy, and that forces a regeneration of the single post page. I've disabled comments on that post and are keep analyzing.

    No Spam Karma 2? I was acutally worried that using Akismet would slow the site down if Akismet got overwhelmed. But now it's MY local database which is overwhelmed, so maybe you're right.

  12. ringmaster
    Posted 10 years ago #

    This might not be entirely helpful for your situation, but over the past few weeks I've been going over some optimization of WordPress for my own needs, and I thought I would share.

    One of the major factors in getting reliable service was keeping the server from dipping too deeply into swap. Having enough memory available to keep swap to a minimum is one of my new priorities with the new virtual server. When the swap started to get used significantly (usually from a torrent of MySQL queries), it began a cascade effect that would bring the server to its knees. Adding more memory to the system solved the issue instantly.

    I installed a PHP opcode cache (APC) on the server. This had a significant impact on load, because the server no longer wastes time recompiling PHP scripts every time they are requested. My pages are running nearly 100% from the cache of compiled PHP, meaning that every page request is using a precompiled script and skipping over the load generated by compilation. Before APC my load was near 3 on average during high traffic times, now it's consistently under 1.

    Another thing that may help performance would be a combination of using the MySQL qcache and a plugin that helps optimize queries for it. You can read about and get the plugin from the wp-hackers mailing list archive. Note that the data from this plugin helped drive development, and that WordPress 2.1 (when it is released) will have this sort of optimization already built in.

    Of course WP-Cache is a decent plugin for creating static caches, but it still might not be enough by itself to offset the issue completely. Also, it doesn't work out-of-the-box with APC. You'll want to review these notes on APC and wp-cache2. Currently, my site does not run both, since I couldn't get them to play well together, but I've been happier with APC's performance than with WP-Cache's.

    The sad, unspoken truth is that you can optimize your code until your ears bleed, but sometimes you just need a bigger box.

  13. WillHines
    Posted 10 years ago #

    Disabled Spam Karma 2. Enabled Akismet.
    Disabled wp-cron.
    Disabled comment on the post which is attracting all the traffic.

    That might have put out the fire.

    I'm going to need to be able to keep comments open while sustaining high traffic, but if it puts out the fire, I'm happy to disable them temporarily.

    Awesome help, everyone!

  14. ek_wals
    Posted 10 years ago #

    Just noticed LightPress - a frontend for WP that claims to greatly increase the speed.

    See: http://forum.textdrive.com/viewtopic.php?id=5716

    Talks about a site that reaches 400 requests/min.

  15. Matt Mullenweg
    Posted 10 years ago #

    For that level of traffic you do not need a 1000/mo server as suggested earlier in the thread.

    Lightpress is different software, it doesn't use WP themes or plugins or...

    Properly configured wp-cache can give you 200+ requests per second, or about 17 million pageviews per day, which is more than enough for Mr Hines' site.

    Owen's advice is sound.

  16. davidchait
    Posted 10 years ago #

    With WP-Cache, WP can certainly handle hundreds of requests per minute on a decent webserver. LightPress might be a bit more optimized in some ways, but behind in features in others.

    Apache performance at some point becomes limiting factor, in which case looking into lighttpd, thpptd, etc., might be useful (if only for image serving...).


  17. Mark (podz)
    Support Maven
    Posted 10 years ago #

    @WillHines - Matt mentions Owen. Owen = ringmaster :)

  18. WillHines
    Posted 10 years ago #

    Yeah, I'm not interested in Lightpress if only because its support community is not as evolved. And I've never heard of it before today. WordPress is great and FEELS right -- I'm going to keep pushing to have it be our main tool, though if I can't enable comments I will lose that battle.

    Thanks, Matt and Owen and everyone for the fast response. I'll look into getting more memory. I'm totally fine with getting a bigger box if I can convince mediatemple to let me(?). I'll also ask them about installing a PHP opcode cache.

  19. manstraw
    Posted 10 years ago #

    I shouldn't have posted the link to the 1000/month server. I wasn't suggesting you get that server, but was trying to make the point that if you want to be a safe, get the best decicated server you can afford. The next time you get a burst of activity, will your optimizations handle it? When you can answer that question with confidence, stop where you are. You're not trying to handle your average traffic, you're trying to handle your bursts.

  20. WillHines
    Posted 10 years ago #

    I've gotten a mandate: we're moving to Movable Type. Sucks! I first discovered WordPress two years ago when I got sick of waiting for MT to publish pages. But now the allure of flat HTML has won out. Not my decision. I guess if the ISP will support this, then it'll be better for me, and I can keep using WP for other projects.

    Thanks, everyone.

  21. davidchait
    Posted 10 years ago #

    That's just silly... but oh well.

    This thread might indicate that the Codex needs some pages on VPS optimizations and performance tuning for large sites.

  22. WillHines
    Posted 10 years ago #

    It's infuriating.

    Lists of optimizations are good. Even better would be a handy list of highly-trafficked, high-profile sites that use WordPress.org. One mention of "huge-site-you-have-heard-of.com uses this" is more powerful than a page of statistics.

  23. Dougal Campbell
    Posted 10 years ago #

    Well, I guess it's too late now, but you *did* verify that WP-Cache was creating cache files and serving them up, right?

  24. WillHines
    Posted 10 years ago #

    I did confirm it, and it was a huge help. Brought the server load down from 8 to about 1 -- thwarting our first burst of traffic.

    But we had two more bursts since then which overwhelmed despite WP-Cache.

    It's when a post is heavily commented that seemed to kill us. The Us readers, when they latch onto a post, comment several times a minute. That, combined with the crush of traffic going to the post ("Secret dinner for Nick and Jessica") -- would do it. I had just pinpointed that as a stress point when I got the word that we had to switch. It's still just a hunch, but by disabling comments on popular posts I'm keeping the site up.

    I said "Ok. It's $200."
    "It's not free."
    (hands over credit card).

    I'm SURE WordPress could have worked. But the ISP, or whoever controls the server, is not on board -- well, then it can't work. I'm sure MT will have trouble, but since it was their suggestion I assume they'll battle through whatever problems surface.

  25. Mark (podz)
    Support Maven
    Posted 10 years ago #

    High Traffic:

    I hope you come back and read that list......

  26. davidchait
    Posted 10 years ago #

    Hmmm, when I just visited the homepage, it was not served from the cache.

    Also, the comment counts certainly mean every time there's a comment, at the least the post and the homepage(possibly) need to be flushed. I haven't looked into the guts of wp-cache to see when it force-flushes other than ever N hours.

    However, that said, the 51 comments on that post were generally every few minutes, rather than a few per minute. That's still MUCH less than the potential hit rate of say 20-50 hits per minute. But again, note, it is just as possible you are getting crushed at the server-resource end if you are spiking to hundreds of connections per minute (i.e., many per second). A slow VPS with low memory could just be sunk.

    Certainly, if you aren't getting static pages served up whenever possible, that'll hurt if you're being dugg or slashdotted, etc.

    On a very heavy site, I'd consider delaying cache updates due to comments until maybe every few minutes (and comment-count updates at the same time).

    But, we're working with imperfect information here. Certainly, even a static HTML system is going to have to refresh stuff every time a comment is added (again, the post and the homepage if the post is on it...).


  27. WillHines
    Posted 10 years ago #

    You're right -- the commnent rate is not as bad as a few a minute (it was a few a minute on the Britney pregnant post, but that was just for a short burst).

    I hate it when people use this forum as tech support, but I'll put this here in case I can save someone else headaches:

    I see this in the source on a single post page:
    <!-- Dynamic Page Served (once) in 13.895 seconds -->
    <!-- Cached page served by WP-Cache -->

    And I see files being created in the wp-content/cache directory. So I think I have wp-cache working.

  28. manstraw
    Posted 10 years ago #

    I guess since you're moving to MT, it doesn't matter, but I thought I'd give you some feedback anyway.

    I'm not getting the first page from the cache. I entered http://usmagazine.com and let it do it's thing. These two were done about 30 seconds apart. This is what was at the bottom of the html for each.

    <!-- Dynamic Page Served (once) in 15.743 seconds -->
    <!-- Dynamic Page Served (once) in 15.032 seconds -->

    No mention of the Cached page, and the times are different.

    And after typing the above, I went back. Still no cache, but somewhat faster:

    <!-- Dynamic Page Served (once) in 4.358 seconds -->

    good luck in wherever you end up.

  29. angsuman
    Posted 10 years ago #

    I have posted an article on optimizing WordPress for highly commented blogs. This will solve your problem.

    Give it a try even if it is with simulated loads. Then maybe you can convince your management back.

    Wish you best of luck.

  30. Mark Jaquith
    WordPress Lead Dev
    Posted 10 years ago #

    My suggestions are pretty much the same as Owen's.

    1. If you're dipping into swap, add RAM.
    2. If Apache's connections are maxed, up the limit
    3. If MySQL is running slow, allocate more RAM to it (see number 1) and:
      • Install the plugin that freezes $now, like Owen said
      • enable MySQL's Qcache
      • Then, re-analyze your queries, and tackle any outliers
    4. APC is pretty much a free speed boost. Give it at least 32MB... maybe as much as 64MB or 96MB if you have a lot of pages.
    5. WP-Cache2 is a similar no-brainer, although do note the caveat that Owen linked
    6. Up the expiration on WP-Cache2 to at least double the default
    7. I've done all this, plus relaced WP's file-based object caching with in-memory caching using APC, and it all runs wicked fast. How much traffic it can handle depends on many things. If you're getting a "slashdot" or "digg" type effect, meaning you have a lot of first-time viewers all hitting the same post, WP-Cache will handle millions of hits a day... as they'll all be getting the same static cached file.

      If your traffic is more spread out, say if someone big linked your front page and a majority is going on to click to other pages (this is rare... the vast majority will hit you once and go on to the next item) or if you simply have a large userbase, you might not to be able to handle quite as much, since many hits won't be cached, especially if people have comment cookies, which means they can't get a generic cached page, but the Qcache and the $now freezing and APC will do a lot to minimize that load. You may be able to handle this on a VPS, but a cheap dedicated server (expect to pay $90-$160 a month) will be able to handle it easily, especially with the modifications that I mentioned.

Topic Closed

This topic has been closed to new replies.

About this Topic