Forum Replies Created

Viewing 15 replies - 151 through 165 (of 390 total)
  • Thread Starter archon810

    (@archon810)

    Actually, there seems to be some sort of a race condition here. If I just keep the two lines that flush rules live and watch the value of wp_options rewrite_rules, it keeps flopping between 72 and 74 values – with 2 JSON API rules and without. So it really depends on timing when I comment them out – it could end up with JSON API rules, or it could end up without.

    Thread Starter archon810

    (@archon810)

    Apologies, I’m an idiot. I was using a dev server which still had v4 running. After the update, there are no strict standards warnings anymore.

    Thread Starter archon810

    (@archon810)

    A few things:

    1. If you want to debug this, the current site exhibiting it 100% is http://hive.androidpolice.com
    2. It periodically happens to the main domain as well. For example, several CSS sheets were just being included as http://cdn.androidpolice.com/http://www.androidpolice.com, such as <link rel='stylesheet' id='ap-css-css' href='http://cdn.androidpolice.com/http://www.androidpolice.com/wp-content/themes/ap1/style.css?cachebust=921323&ver=3.9.1' type='text/css' media='all' />. This is a much more serious, production-breaking bug.
    Thread Starter archon810

    (@archon810)

    Frederick, I think you meant to respond here instead of http://wordpress.org/support/topic/suggested-changes-to-object-caching-and-transients?replies=3? That thread could still use a proper reply as well if you have time.

    Thread Starter archon810

    (@archon810)

    Sure you’re replying to the right post, Frederick?

    archon810

    (@archon810)

    Indeed, that’s another fix that should resolve the underlying issue of mismatching params.

    Thread Starter archon810

    (@archon810)

    Just for fun, guess when the object caching bug was fixed in our setup over at androidpolice.com.

    db server: http://i.imgur.com/yJ0wwhk.png
    web server 1: http://i.imgur.com/RFQR4tL.png http://i.imgur.com/EjH2xcT.png

    archon810

    (@archon810)

    I have diagnosed and resolved the issue. Read the write-up here: http://wordpress.org/support/topic/self-diagnosed-and-fixed-w3-total-cache-bug-in-faulty-object-caching.

    This fixes WP-Cron getting stuck and everything else related to object caching not retaining values and creating a crazy amount of unnecessary writes to cache which result in frequent evictions.

    archon810

    (@archon810)

    Thread Starter archon810

    (@archon810)

    Now for part 2.

    Relevant files:

    • lib/W3/Cache/Base.php
    • lib/W3/Cache/Memcached.php
    • lib/W3/Cache/ObjectCache.php

    The solution above basically means that only 2 buckets with keys ” (blank) and 0 will exist. The cached data will be split between these 2 buckets:

    [20-Apr-2014 00:42:20 UTC] APDEBUG setting key version for group  to 44
    [20-Apr-2014 00:42:20 UTC] APDEBUG setting key version for group 0 to 14
    [20-Apr-2014 00:42:20 UTC] APDEBUG setting key version for group  to 44
    [20-Apr-2014 00:42:20 UTC] APDEBUG setting key version for group 0 to 14

    That seems to work, but the intention of the code that deals with object caching is to make the buckets correspond to groups (like ‘options’, ‘transient’, etc), at least that’s what I gather from reading the code.

    The main problem is that the $group value isn’t passed from ObjectCache.php to the respective plugins, such as Apc.php, Memcached.php, etc. This means that the groups for all objects will be set to the defaults, hence 0 and 1 (see part 1 above for why we have the discrepancy in the first place).

    I think the right fix is to also pass the $group value along as well. I’m not sure my patch is 100% complete – it should still be checked by a core developer, but the idea is to do this:

    Index: lib/W3/ObjectCache.php
    ===================================================================
    --- lib/W3/ObjectCache.php      (revision 891523)
    +++ lib/W3/ObjectCache.php      (working copy)
    @@ -164,7 +164,7 @@
                       !in_array($group, $this->nonpersistent_groups) &&
                         $this->_check_can_cache_runtime($group)) {
                 $cache = $this->_get_cache(null, $group);
    -            $v = $cache->get($key);
    +            $v = $cache->get($key, $group);
                 if (is_array($v) && $v['content'] != null)
                     $value = $v['content'];
                 else
    @@ -234,7 +234,7 @@
    
             $this->cache[$key] = $data;
    
    -        if ($this->_caching &&
    +        if ($this->_caching &&
                     !in_array($group, $this->nonpersistent_groups) &&
                     $this->_check_can_cache_runtime($group)) {
                 $cache = $this->_get_cache(null, $group);
    @@ -251,7 +251,7 @@
    
                 $v = array('content' => $data);
                 return $cache->set($key, $v,
    -                ($expire ? $expire : $this->_lifetime));
    +                ($expire ? $expire : $this->_lifetime), $group);
             }
    
             return true;
    @@ -277,7 +277,7 @@
             if ($this->_caching && !in_array($group, $this->nonpersistent_groups)) {
                 $cache = $this->_get_cache(null, $group);
    
    -            return $cache->delete($key);
    +            return $cache->delete($key, $group);
             }
    
             return true;

    At this point, going back to the log produces something like:

    [20-Apr-2014 01:00:39 UTC] APDEBUG setting key version for group post_tag to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group posts to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group category to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group post_tag_relationships to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group category_relationships to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group post_format_relationships to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group post_meta to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group users to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group userlogins to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group useremail to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group userslugs to
    [20-Apr-2014 01:00:40 UTC] APDEBUG setting key version for group user_meta to
    [20-Apr-2014 01:07:45 UTC] APDEBUG setting key version for group default to 15
    [20-Apr-2014 01:07:45 UTC] APDEBUG setting key version for group options to 15
    [20-Apr-2014 01:07:46 UTC] APDEBUG setting key version for group transient to 15

    I’m too tired to figure out if it’s a good or a bad thing that the key versions are the same for different groups, but at least they seem to be consistent across get() and set().

    I’m hoping Frederick, who I’ve been trying to reach to work on this bug, or someone from the W3TC dev team will see this, respond, and evaluate.

    Thank you and good luck, everyone.

    archon810

    (@archon810)

    I don’t think this is correct, actually. object-cache.php doesn’t call w3-total-cache/lib/W3/Cache/Apc.php directly. In fact, it first calls ObjectCache.php, and only from there does it call to Apc.php, Memcached, php, etc.

    The order of params is actually correct once you look at set() in w3-total-cache/lib/W3/ObjectCache.php.

    I did, however, just spent days on figuring out why object caching doesn’t work reliably, and I have figured it out after much blood, sweat, and tears.

    I’m going to post it separately, drop a link here, and attempt to get in contact with Frederick once again.

    archon810

    (@archon810)

    I take this back, the problem has come back, even though I now have 4x512MB with 4K allowed connections. And it’s not just limited to cron – regular transients aren’t being returned either.

    According to memcached stats via telnet, each server is only serving about 100 connections, and is definitely accepting more.

    I’m at a loss once again.

    Understood.

    In my case, memcached has 3x 512MB servers allocated for a total of 1.5GB, which is quite a bit.

    @jochent The eviction counter is not really an indication of a problem – it simply means older data will be ejected, but data just put in recently shouldn’t be (unless the memory allocated to memcached is really tiny).

    In fact, if the server is running for a really long time, and the data being written doesn’t use the same keys (it all depends on your plugins and theme code), eventually older data will be evicted, and that’s fine in theory most of the time, though that also means that things like transients that are meant to stick around for a long time may disappear before it’s time for them to, which would have happened with persistent storage.

    The real issue in my case was the number of allowed connections, which on a busy server was simply outpaced by the number of threads connecting to the memcached server.

    I’m not 100% sure, but for me, switching object caching to APC, for example, sorted the issue instantly, then back to memcached, and it’s broken. Not until I figured out the memcached servers were seemingly misbehaving, at least in my case.

Viewing 15 replies - 151 through 165 (of 390 total)