WordPress.org

Ready to get started?Download WordPress

Forums

Search Engine
[resolved] Search Engine not following robot.txt/noindex rules (7 posts)

  1. robanna
    Member
    Posted 2 years ago #

    I have a robots text file http://apexprd.org/robots.txt and meta nofollow and noindex on pages that are bing indexed.

    my _searchengine_template table has disable_robots = 0. Is that correct? Shouldn't it be skipping those pages with noindex?

    Thanks-

    http://wordpress.org/extend/plugins/search-engine/

  2. Scott Kingsley Clark
    Member
    Plugin Author

    Posted 2 years ago #

    The robots.txt parser is not that advanced yet, and I was looking for your nofollow/noindex in robots tag on a few of the pages but didn't see it, which pages have it set so I can test?

  3. Scott Kingsley Clark
    Member
    Plugin Author

    Posted 2 years ago #

    As a solution for not indexing certain URIs, you could try using the blacklist setting in the Search Index Template advanced section of the Wizard when creating a new template. If you've already indexed, you can go to Search Engine >> Settings >> Reset to clear out your data and start over

  4. robanna
    Member
    Posted 2 years ago #

    Basically every page/post has a print version so my whole site is being index twice; once for the regular page and once for the print version. It's also indexing the shortcode version even though the robot text asks it not to. So, our website with 1000+ page now is trying to index 3000+ pages with 3 of the same search results for each query.

    noindex, nofollow example: http://apexprd.org/home/printpage/

  5. robanna
    Member
    Posted 2 years ago #

    I'll give the blacklists a try. Thanks-

  6. Scott Kingsley Clark
    Member
    Plugin Author

    Posted 2 years ago #

    It also might be worth trying to set your meta tag's name to "robots" instead of "Robots", I don't think I had a strtolower set on that loop, I just added it to my dev copy.

  7. robanna
    Member
    Posted 2 years ago #

    Thanks, I never gave it a second thought.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic