WordPress.org

Ready to get started?Download WordPress

Forums

WordPress SEO by Yoast
Request to add noindex to meta robots tag on 404 pages (6 posts)

  1. jp2112
    Member
    Posted 2 years ago #

    WordPress version: 3.3.2
    WordPress SEO version: 1.1.9

    I notice that search pages (is_search) come back with the correct meta robots tag:

    <meta name="robots" content="noindex,follow,noodp,noydir"/>

    Example search result page: http://www.jpsoftwaretech.com/search/xmlhttp

    However the 404 pages come back with this meta robots tag:

    <meta name="robots" content="noodp,noydir"/>

    Example 404 page: http://www.jpsoftwaretech.com/dgfcegfebc

    Notice the lack of 'noindex'. I've taken to adding a second meta robots tag like this:

    // noindex 404 pages
    add_action('wp_head', 'stopindexing404pages');
    function stopindexing404pages() {
      if (is_404()) {
        echo '<meta name="robots" content="noindex, follow">';
      }
    }

    Obviously I would prefer the plugin do this, as I'm not sure if the search engines can understand two meta robots tags on a single page. I can't imagine any circumstance in which someone would want a search engine to index 404 pages, so it seems to me this should be the default behavior.

    http://wordpress.org/extend/plugins/wordpress-seo/

  2. Pann
    Member
    Posted 2 years ago #

    Search engines won't index any 404 pages since wordpress sends a 404 header along with the content, spiders are clever enough to stop right there, every content in that page is just to ease the blow on the end user:

    Request URL:http://www.jpsoftwaretech.com/dgfcegfebc
    Request Method:GET
    Status Code:404 Not Found

  3. jp2112
    Member
    Posted 2 years ago #

    OK, I think I actually figured this out on my own *crossing fingers*

    In the function called robots(), change

    } else {
                if ( is_search() ) {
                    $robots['index']  = 'noindex';
                    $robots['follow'] = 'follow';

    to this:

    } else {
                if ( is_search() || is_404() ) {
                    $robots['index']  = 'noindex';
                    $robots['follow'] = 'follow';

    And now 404 and search pages are marked as noindex in the meta robots tag!

    Joost, is there anything I can do to make sure this gets included in the next update?

  4. jp2112
    Member
    Posted 2 years ago #

    @Pann -- thank you. I'm afraid I can't leave it up to them like that. I got slammed by Google's last few updates so I'm trying everything I can to make sure my site is being properly indexed.

  5. Pann
    Member
    Posted 2 years ago #

    It most certainly won't hurt, but they are much more likely to obey the response header than a meta tag :-)

    With that said, I do remember Cutts saying that turning 404 pages into honeycombs and not letting users adequately aware that the thing they requested was actually *not found*, is bad for SEO.

  6. jp2112
    Member
    Posted 2 years ago #

    turning 404 pages into honeycombs and not letting users adequately aware that the thing they requested was actually *not found*, is bad for SEO.

    I'm confused, does that have anything to do with what I'm trying to accomplish? I'm just trying to get the plugin to show the same meta robots tag for search and 404 pages.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic