WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] Prevent indexing of feed pages? (13 posts)

  1. Hostwolf
    Member
    Posted 7 years ago #

    How do I do this. I had some keyword traffic to this becoming a webmaster for dummies post's comment feed.

    http://www.google.com/search?q=becoming+a+webmaster&hl=en&start=10&sa=N

    Shows you that first the post comes up and then the feed page . The feed page doesn't add any value whatsoever and will turn visitors off. How do I prevent their indexing?

  2. lelion
    Member
    Posted 7 years ago #

    Yes, I'd like to know the same... feeds are ok for rss readers, but for google, .php pages are ok ;-)

    Anyone? :)

  3. Hostwolf
    Member
    Posted 7 years ago #

    well basically the answer I came up with myself is to add this to the robots.txt

    Disallow: /blog/feed/
    Disallow: /blog/comments/feed/

    But there are still other spots for spiders to access your feeds. I am going to post about removing them.

  4. lelion
    Member
    Posted 7 years ago #

    That would work, yes, what you suggest, but would it work also for the feeds for every one post and every one comments thread WP automatically generates?...

    URLs indexed by google like:
    http://www.domain.com/wordpress/2007/01/15/title-of-post/feed/

    ???

  5. Hostwolf
    Member
    Posted 7 years ago #

    That what I meant by

    But there are still other spots for spiders to access your feeds. I am going to post about removing them.

    Those are the other spots for the spiders to access feeds. I think one needs to remove these in the php source code but I'm not sure whether that would cause trouble with upgrading in the future.

    I can't see how to solve this solely with robots.txt

  6. Hostwolf
    Member
    Posted 7 years ago #

    btw the example url you posted are the exact urls I want to get out of the search engines. They are nothing but pollution.

  7. lelion
    Member
    Posted 7 years ago #

    Yes, I think so, too. The feed example I posted contain SAME text as the posts, so why google should index it?

    "" I can't see how to solve this solely with robots.txt ""

    I can't, too, but would like too... Maybe a rule should be created, which will filter everything ending with "/feed"?

    Anyone any ideas on that? :)

  8. Hostwolf
    Member
    Posted 7 years ago #

    Meanwhile I found out that this can actually be done. It doesn't conform to the robots.txt standards but Google and Slurp accept it so that's as good as it gets.

    I just implemented

    User-Agent: *
    Disallow: */feed/

  9. lelion
    Member
    Posted 7 years ago #

    Thx, great, nice tip!

    How can you say it'll work?

    I have robots.txt and may implement it today or tomorrow...

    (Would be nice also for others to know...)

    If you're sure it'll work, we may mark this example as "resolved" and help others as well, right? :-)

  10. Hostwolf
    Member
    Posted 7 years ago #

    There is plenty of evidence that this works if you read these SERPS. People at sitepoint also confirmed this.

  11. lelion
    Member
    Posted 7 years ago #

    Thx! I'll implement this tomorrow, then, and will check what will happen...

    PS Going to read now... SERPS, of course:)

  12. lelion
    Member
    Posted 7 years ago #

    Just implemented this trick

    User-Agent: *
    Disallow: */feed/

    ...and now waiting to see what'll happen:)

  13. che1959
    Member
    Posted 6 years ago #

    I just saw this and wonder, why would you do this?

    The * means any user agent. That also means bots that access your feed. Wouldn't it mean that the technorati bot couldn't read your feed? What about other sites that read your feed to what has changed?

    Why not just specifically name googlebot and slurp by name?

Topic Closed

This topic has been closed to new replies.

About this Topic