Support » Everything else WordPress » Questions about Google and the robots.txt file…

  • I have followed the guide on this site about SEO for wordpress and I will say that I am surprised on how well it does work… but I have a concern…

    I followed the guide on how to setup the robot.txt (recommended setup for a wordpress site) and I see that it wants you to block things like posts and categories so that you don’t let google see duplicate content from… say …. your RSS feed.

    My issue with this is that i studies my site stats and I see what people are searching for on google… and it’s great that my site comes up and even the information from the relevant post is under the link in google, but the link to the page is just my home page… not the relevant post.

    I assume this is because of the robot.txt blocking google from seeing these directories… would it really hurt my rank to unblock these directories if I have a feedburner account? Anyone have any other suggestions about the robot.txt file.

    Thanks for any help.

    – James

Viewing 6 replies - 1 through 6 (of 6 total)
  • whooami



    I dont keep google out of my feeds, or any other post content, for that matter. I have seen no detrimental impact.

    My pagerank doesnt suffer one bit.

    I take that back — The only thing I restrict googlebot from doing with WP is looking at printable pages, other than that, I let it read whatever “content” it wants.

    I don’t block Google for anything and have noticed no problems. I don’t think duplicate content on the same site is going to have any significant detrimental value. On multiple sites, now that could be an issue.

    And all my pages are “printable”, although I do it differently than whooami does it. 😉

    Could you guys show me an example of your robots.txt? I just copied and pasted the example shown…

    Here’s mine:

    User-agent: ia_archiver
    Disallow: /

    I used that simply because I don’t want my pages on Otherwise, I’d leave it completely blank. Blank = allowed.

    The only thing you can do with a robots.txt is block. If you don’t want to block, leave it blank or don’t have it at all.

    can you share with me the link to guide on this site about SEO for wordpress? Thanks.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Questions about Google and the robots.txt file…’ is closed to new replies.