Support » Plugin: WP Super Cache » x-robot-tag lost when caching

  • So i have an issue with this plugin.

    I have various pages with an x-robots-tag header set with “noindex,noarchive,nosnippet,follow” to stop indexing of various pages.

    This is done in 2 ways, for some things this is done directly in the template file with a header command – to stop indexing of a whole host of junk pages such as tag pages for instance.

    Some are setup using SEO Ultimate plugin for wordpress.

    Doesnt matter with either approach, it appears once the page is cached and that cache is served the page no longer reports an x-robots-tag header at all.
    I have the option enabled in settings “Cache HTTP headers with page content.”

    How can this be fixed??

    The page I need help with: [log in to see the link]

Viewing 6 replies - 1 through 6 (of 6 total)
  • Saša

    (@stodorovic)

    I’ve similar issue on a installation with Apache 2.2/FCGI, but it works on my other installations. Function apache_response_headers returns empty array. Also, function flush can’t be called from output buffer handlers. So, it’s issue related to Apache and PHP handlers (not related to WP/WPSC).

    Sending headers in template files isn’t good solution. Also html content should have proper meta robots tag. Header X-Robots is needed only for XML (eg. feeds), PDF and similar files when isn’t possible to use meta robots.

    Better way is to set noindex (example is for SEO Ultimate plugin) via filter (in functions.php):

    add_filter( 'su_meta_robots', 'my_meta_robots', 20 );
    function my_meta_robots( $commands ) {
    	if ( is_tag() ) {
    		return array( 'noindex', 'nofollow' );
    	}
    	
    	return $commands;
    }
    
    // Adds custom X-Robots-Tag before writing wp-cache file.
    add_filter( 'wp_cache_meta', 'my_cache_robots_tag' );
    function my_cache_robots_tag( $wp_cache_meta ) {
            if ( isset( $GLOBALS[ 'wp_super_cache_query' ][ 'is_feed' ] ) ) {
                    $wp_cache_meta[ 'headers' ][ 'X-Robots-Tag' ] = 'X-Robots-Tag: noindex,follow';
            }
            
            return $wp_cache_meta;
    }
    

    I hope that helps. With this code you can use expert mode and avoid “Cache HTTP headers with page content”.

    as much as I appreciate your reply and I do agree what im doing manually is a little “sloppy” it is not my main concern right now.

    As I said even noindex set by SEO Ultimate is not working once a page is cached.

    So I need to fix this issue before I can move on with any other improvements – it looks like your function can do my custom rewrites, but does not handle the manual ones done through SEO Ultimate… So this only solves part of a big problem.

    Also I do not have your issue, I’m not using FCGI which I’ve had dealings with in the past, FCGI is terrible and makes allot of advanced things hard… Anyway apache_response_headers works fine in my setup. SUPHP/PHP 7.2

    Ok I removed WP Super Cache and installed W3 Total Cache instead. Set it all up turned it on and guess what…

    Not only is it caching the pages, its recognising the headers and caching those too. So this confirms that there is no issue seeing the headers as claimed.

    So until this issue is fixed (because i would prefer to use WPSC) I will have to use W3 Total Cache.

    Saša

    (@stodorovic)

    I’ve found bug in PHP 7.x related to function apache_response_headers (php 5.6 works). It’s related to CGI SAPI (suPHP uses it). Last character is removed from all headers. I see issue in PHP source code and I’ll report it.

    Related issue on github – Cache HTTP Headers with page content fails in some cases. You could follow this issue for more details.

    Yes I noticed the same issue on our server using PHP 7.2 and suPHP. You are correct that function is losing the last character from the header names.

    Initially I wondered if this was the issue for WPSC, however it doesnt even restore bad headers (missing last character) so this leads me to believe it just isnt working at all, unless it only restores select headers it dictates, which would open up multiple other problems with all the various headers that people might use and might be considered vital.

    Saša

    (@stodorovic)

    It’s strange why nobody reported this bug. Anyway, I’ve created php/php-src#3566 which fixed it and I’ve added tests for function apache_response_headers. So, it’ll be fixed in future versions of PHP.

    From other side, WPSC doesn’t need to use apache_response_headers because it’s possible to use more general function headers_list and I’ve optimized PHP code in the function wp_cache_get_response_headers(to keep performance and improve portability) – PR #618.

    After merging both PRs, this issue will be properly fixed on both sides (PHP 7.x and WPSC). In meantime, you can update WPSC with PR #618. It’ll be helpful if you test this PR and send feedback.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘x-robot-tag lost when caching’ is closed to new replies.