WordPress.org

Ready to get started?Download WordPress

Forums

Robot.txt Question/Issue (1 post)

  1. matthewjahn
    Member
    Posted 2 years ago #

    Hey all,

    So I'm new to robot.txt, and have a question;

    I read that it's good SEO to add the following code to your robot.txt file;

    User-agent: *
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /cgi-bin
    Disallow: /search
    Disallow: /feed
    Disallow: /rss
    Disallow: /comments/feed
    Disallow: /feed/$
    Disallow: /*/feed/$
    Disallow: /*/feed/rss/$
    Disallow: /*/trackback/$

    OK, so I didn't really know how to do that, but luckily I found a place to enter this with the "SEO Ultimate" plugin. Easy, you just go to the File Editor section and enter the code into the correct field and it loads the code to your robot.txt file. I checked the file at http://www.example.com/robot.txt and all looked like it was supposed to.

    Then I went to do the same thing on my blog. I have a separate WP installation on a subdirectory like http://www.example.com/blog. Now, I also read for this, the following code should be placed;

    User-agent: *
    Disallow: /blog/wp-admin
    Disallow: /blog/wp-includes
    Disallow: /blog/wp-content/plugins
    Disallow: /blog/wp-content/cache
    Disallow: /blog/wp-content/themes
    Disallow: /blog/trackback
    Disallow: /cgi-bin
    Disallow: /search
    Disallow: /blog/feed
    Disallow: /blog/rss
    Disallow: /blog/comments/feed
    Disallow: /blog/feed/$
    Disallow: /blog/*/feed/$
    Disallow: /blog/*/feed/rss/$
    Disallow: /blog/*/trackback/$

    Now, the problem is, when I do this on the blog, and try to look up my http://www.example.com/blog/robot.txt file, I get a 404 error instead of the correct script.

    Should I be putting the above code into the http://www.example.com/robot.txt file and not the subdirectory?

    What am I doing wrong?

    Any help would be great!

    - Matthew

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags