Robots.txt

  • Resolved bobjones

    (@bobjones)


    12 years, 11 months ago

    I’ve set my robots.txt settings in WordPress SEO. But I also noticed the robots.txt page in the root folder of my site is blank.

    Should I leave the blank robots.txt? Is WordPress SEO handling those settings in another way?

    Meanwile, my site has been active for a month and has not yet been indexed. So, I want to make sure everything is OK with robots.txt.

    Thanks.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Contributor Joost de Valk

    (@joostdevalk)

    12 years, 11 months ago

    You don’t need a robots.txt, my plugin doesn’t do to much with it either.

    Thread Starter bobjones

    (@bobjones)

    12 years, 11 months ago

    You don’t need a robots.txt?

    There are endless discussions on the web of what’s appropriate for robots.txt. Now you say we don’t need it at all?

    When did this new orthodoxy take hold?

    By the way, if your plugin isn’t editing robots.txt, how does it determine whether items such as tags and categories get indexed or not? I understood that to be the job of robots.txt.

    Plugin Contributor Joost de Valk

    (@joostdevalk)

    12 years, 11 months ago

    No it’s not the job of robots.txt, it’s the job of the robots meta tag on each page. You should read this article and watch the video:

    http://yoast.com/prevent-site-being-indexed/

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Robots.txt’ is closed to new replies.