Where do I put my robots.txt file for WP?

  • jdquel

    (@jdquel)


    17 years, 2 months ago

    Forgive me if this question is a little basic, but I am relatively new to using robots.txt.

    Is it possible to allow Google and other search engines to list your site without allowing them to index your actual content in their cache?

    Also, would robots.txt for a wordpress blog go in the public_html folder or somewhere else?

    Thanks!

Viewing 2 replies - 1 through 2 (of 2 total)
  • Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    17 years, 2 months ago

    There is no way to disable caching using robots.txt. You have to use a meta tag in each page’s header. Just add this to your theme’s header.php file (or wherever your theme has its <head> section:
    <meta name="robots" content="noarchive" />

    This will prevent Google and Yahoo and other nice crawlers from offering a cached copy of your content. However, they must have your content to index it. So they will have it in their database, they just won’t offer it for viewing by others.

    If you want Yahoo and others to have the content available and to only block Google, use this instead:
    <meta name="googlebot" content="noarchive" />

    As for the location of robots.txt, it must always be located in the site root. Probably your public html folder, although that’s not guaranteed. Wherever you put it, it must be found at http://example.com/robots.txt (replace example.com with your domain name).

    Thread Starter jdquel

    (@jdquel)

    17 years, 2 months ago

    This helps alot. Thank you much!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Where do I put my robots.txt file for WP?’ is closed to new replies.