WordPress.org

Ready to get started?Download WordPress

Forums

Questions about Google and the robots.txt file... (7 posts)

  1. jderosa3
    Member
    Posted 6 years ago #

    I have followed the guide on this site about SEO for wordpress and I will say that I am surprised on how well it does work... but I have a concern...

    I followed the guide on how to setup the robot.txt (recommended setup for a wordpress site) and I see that it wants you to block things like posts and categories so that you don't let google see duplicate content from... say .... your RSS feed.

    My issue with this is that i studies my site stats and I see what people are searching for on google... and it's great that my site comes up and even the information from the relevant post is under the link in google, but the link to the page is just my home page... not the relevant post.

    I assume this is because of the robot.txt blocking google from seeing these directories... would it really hurt my rank to unblock these directories if I have a feedburner account? Anyone have any other suggestions about the robot.txt file.

    Thanks for any help.

    - James

  2. whooami
    Member
    Posted 6 years ago #

    I dont keep google out of my feeds, or any other post content, for that matter. I have seen no detrimental impact.

    My pagerank doesnt suffer one bit.

    I take that back -- The only thing I restrict googlebot from doing with WP is looking at printable pages, other than that, I let it read whatever "content" it wants.

  3. Samuel Wood (Otto)
    Tech Ninja
    Posted 6 years ago #

    I don't block Google for anything and have noticed no problems. I don't think duplicate content on the same site is going to have any significant detrimental value. On multiple sites, now that could be an issue.

    And all my pages are "printable", although I do it differently than whooami does it. ;)

  4. jderosa3
    Member
    Posted 6 years ago #

    Could you guys show me an example of your robots.txt? I just copied and pasted the example shown...

  5. Samuel Wood (Otto)
    Tech Ninja
    Posted 6 years ago #

    Here's mine:

    User-agent: ia_archiver
    Disallow: /

    I used that simply because I don't want my pages on archive.org. Otherwise, I'd leave it completely blank. Blank = allowed.

    The only thing you can do with a robots.txt is block. If you don't want to block, leave it blank or don't have it at all.

  6. matwolf
    Member
    Posted 6 years ago #

    can you share with me the link to guide on this site about SEO for wordpress? Thanks.

  7. jderosa3
    Member
    Posted 6 years ago #

Topic Closed

This topic has been closed to new replies.

About this Topic