WordPress.org

Support

Support » Plugins and Hacks » Search Engine » [Resolved] [Plugin: Search Engine] Search Engine not following robot.txt/noindex rules

[Resolved] [Plugin: Search Engine] Search Engine not following robot.txt/noindex rules

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Author Scott Kingsley Clark

    @sc0ttkclark

    The robots.txt parser is not that advanced yet, and I was looking for your nofollow/noindex in robots tag on a few of the pages but didn’t see it, which pages have it set so I can test?

    Plugin Author Scott Kingsley Clark

    @sc0ttkclark

    As a solution for not indexing certain URIs, you could try using the blacklist setting in the Search Index Template advanced section of the Wizard when creating a new template. If you’ve already indexed, you can go to Search Engine >> Settings >> Reset to clear out your data and start over

    Basically every page/post has a print version so my whole site is being index twice; once for the regular page and once for the print version. It’s also indexing the shortcode version even though the robot text asks it not to. So, our website with 1000+ page now is trying to index 3000+ pages with 3 of the same search results for each query.

    noindex, nofollow example: http://apexprd.org/home/printpage/

    I’ll give the blacklists a try. Thanks-

    Plugin Author Scott Kingsley Clark

    @sc0ttkclark

    It also might be worth trying to set your meta tag’s name to “robots” instead of “Robots”, I don’t think I had a strtolower set on that loop, I just added it to my dev copy.

    Thanks, I never gave it a second thought.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘[Resolved] [Plugin: Search Engine] Search Engine not following robot.txt/noindex rules’ is closed to new replies.
Skip to toolbar