Support » Plugin: Yoast SEO » [Plugin: WordPress SEO by Yoast] Request to add noindex to meta robots tag on 404 pages

  • WordPress version: 3.3.2
    WordPress SEO version: 1.1.9

    I notice that search pages (is_search) come back with the correct meta robots tag:

    <meta name="robots" content="noindex,follow,noodp,noydir"/>

    Example search result page: http://www.jpsoftwaretech.com/search/xmlhttp

    However the 404 pages come back with this meta robots tag:

    <meta name="robots" content="noodp,noydir"/>

    Example 404 page: http://www.jpsoftwaretech.com/dgfcegfebc

    Notice the lack of ‘noindex’. I’ve taken to adding a second meta robots tag like this:

    // noindex 404 pages
    add_action('wp_head', 'stopindexing404pages');
    function stopindexing404pages() {
      if (is_404()) {
        echo '<meta name="robots" content="noindex, follow">';
      }
    }

    Obviously I would prefer the plugin do this, as I’m not sure if the search engines can understand two meta robots tags on a single page. I can’t imagine any circumstance in which someone would want a search engine to index 404 pages, so it seems to me this should be the default behavior.

    http://wordpress.org/extend/plugins/wordpress-seo/

Viewing 5 replies - 1 through 5 (of 5 total)
  • Search engines won’t index any 404 pages since wordpress sends a 404 header along with the content, spiders are clever enough to stop right there, every content in that page is just to ease the blow on the end user:

    Request URL:http://www.jpsoftwaretech.com/dgfcegfebc
    Request Method:GET
    Status Code:404 Not Found

    OK, I think I actually figured this out on my own *crossing fingers*

    In the function called robots(), change

    } else {
                if ( is_search() ) {
                    $robots['index']  = 'noindex';
                    $robots['follow'] = 'follow';

    to this:

    } else {
                if ( is_search() || is_404() ) {
                    $robots['index']  = 'noindex';
                    $robots['follow'] = 'follow';

    And now 404 and search pages are marked as noindex in the meta robots tag!

    Joost, is there anything I can do to make sure this gets included in the next update?

    @pann — thank you. I’m afraid I can’t leave it up to them like that. I got slammed by Google’s last few updates so I’m trying everything I can to make sure my site is being properly indexed.

    It most certainly won’t hurt, but they are much more likely to obey the response header than a meta tag 🙂

    With that said, I do remember Cutts saying that turning 404 pages into honeycombs and not letting users adequately aware that the thing they requested was actually *not found*, is bad for SEO.

    turning 404 pages into honeycombs and not letting users adequately aware that the thing they requested was actually *not found*, is bad for SEO.

    I’m confused, does that have anything to do with what I’m trying to accomplish? I’m just trying to get the plugin to show the same meta robots tag for search and 404 pages.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘[Plugin: WordPress SEO by Yoast] Request to add noindex to meta robots tag on 404 pages’ is closed to new replies.