WordPress.org

Ready to get started?Download WordPress

Forums

WordPress SEO by Yoast
[resolved] Robots.txt (4 posts)

  1. bobjones
    Member
    Posted 3 years ago #

    I've set my robots.txt settings in WordPress SEO. But I also noticed the robots.txt page in the root folder of my site is blank.

    Should I leave the blank robots.txt? Is WordPress SEO handling those settings in another way?

    Meanwile, my site has been active for a month and has not yet been indexed. So, I want to make sure everything is OK with robots.txt.

    Thanks.

  2. Joost de Valk
    Member
    Plugin Author

    Posted 3 years ago #

    You don't need a robots.txt, my plugin doesn't do to much with it either.

  3. bobjones
    Member
    Posted 3 years ago #

    You don't need a robots.txt?

    There are endless discussions on the web of what's appropriate for robots.txt. Now you say we don't need it at all?

    When did this new orthodoxy take hold?

    By the way, if your plugin isn't editing robots.txt, how does it determine whether items such as tags and categories get indexed or not? I understood that to be the job of robots.txt.

  4. Joost de Valk
    Member
    Plugin Author

    Posted 3 years ago #

    No it's not the job of robots.txt, it's the job of the robots meta tag on each page. You should read this article and watch the video:

    http://yoast.com/prevent-site-being-indexed/

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic