WordPress.org

Ready to get started?Download WordPress

Forums

Robots.txt (15 posts)

  1. krbmedia
    Member
    Posted 7 years ago #

    Is there an easy way to create this file for WP so Google does not access any sensitive areas?

  2. Samuel Wood (Otto)
    Tech Ninja
    Posted 7 years ago #

    Define "easy". If you mean "create it in a text editor and put it on your site", then yeah, that's pretty easy. :)

  3. Chris_K
    Member
    Posted 7 years ago #

    Notepad?

  4. pizdin_dim
    Member
    Posted 7 years ago #

    Notetab is much better.

  5. Samuel B
    moderator
    Posted 7 years ago #

    Put a robots.txt file in your WP root.
    Something like:
    User-agent: *
    Disallow: /aboutme/
    Disallow: /stats/

  6. krbmedia
    Member
    Posted 7 years ago #

    Do I need to worry about the admin area?

  7. Samuel B
    moderator
    Posted 7 years ago #

    Up to you. I don't have my whole wp-admin folder indexed.

  8. krbmedia
    Member
    Posted 7 years ago #

    So would it be like so?

    User-agent: *
    Disallow: /aboutme/
    Disallow: /stats/
    Disallow: /wp-admin/

  9. Samuel B
    moderator
    Posted 7 years ago #

    Yes exactly (though "stats" and "aboutme" were examples)
    If you want to do a specific file:
    User-agent: *
    Disallow: /aboutme/
    Disallow: /stats/
    Disallow: /wp-admin/
    Disallow: /wp-includes/index.php

  10. KatGirl
    Member
    Posted 7 years ago #

    is there something in the codex about this for indepth reading?

  11. Chris_K
    Member
    Posted 7 years ago #

    Google has lots of links for robots.txt. It certainly isn't a WordPress only thing -- it applies to all web sites. :-)

  12. Samuel Wood (Otto)
    Tech Ninja
    Posted 7 years ago #

  13. KatGirl
    Member
    Posted 7 years ago #

    Thanks for the link. I'll check it out.

  14. davidpey
    Member
    Posted 7 years ago #

    Hi all!

    Sorry for the late bump of this... but I have a domain in which I have a Joomla installation at / and WordPress at /blog. I only have 1 robots.txt file in / and here is what is currently in it:

    *****************
    User-agent: *
    Allow: /sitemap.xml
    Allow: /blog/sitemap.xml
    Allow: /index.php
    Allow: /blog/index.php
    Disallow: /*.php$
    Disallow: /blog/*.php$

    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.php-dist$
    Disallow: /*.ico$
    Disallow: /blog/*.js$
    Disallow: /blog/*.inc$
    Disallow: /blog/*.css$
    Disallow: /blog/*.wmv$
    Disallow: /blog/*.cgi$
    Disallow: /blog/*.xhtml$
    Disallow: /blog/*.php-dist$
    Disallow: /blog/*.ico$
    Disallow: /administrator/
    Disallow: /cache/
    Disallow: /components/
    Disallow: /editor/
    Disallow: /help/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /language/
    Disallow: /mambots/
    Disallow: /media/
    Disallow: /modules/
    Disallow: /templates/
    Disallow: /installation/
    Disallow: /cgi-bin/
    Disallow: /blog/wp-content/
    Disallow: /blog/wp-admin/
    Disallow: /blog/wp-includes/

    Please take a look at the items highlighted in bold (particularly the first few)--and let me know if this is allowed (and especially if it will work) in the robots.txt file?

    Also, should I have a separate robots.txt file for the /blog directory?

    Many thanks!

    David

  15. whooami
    Member
    Posted 7 years ago #

    1. you can only have ONE robots.txt and it should me at the root of your domain. Robots will not look anywhere else for it.

    2. You dont need the "allows"

    As far as this:

    Allow: /blog/index.php
    Disallow: /*.php$ <-- that blocks index.php, archives.php, etc.. WHY are you doing that? Joomla and WordPress are both PHP based apps.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.