WordPress.org

Ready to get started?Download WordPress

Forums

Am I using an incorrect configuration in robots.txt (3 posts)

  1. TheVirtualPreacher
    Member
    Posted 3 years ago #

    Hi,

    I am not sure if I am using a correctly configured robots.txt in my WordPress installation.

    Google Web Master tools is reporting that the configuration prevents Google from indexing my "tags"

    I see that it's blocking Google from indexing my categories. Is this a healthy SEO practice???

    This is my current configuration,

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: http://www.virtualpreacher.org/sitemap.xml

  2. Rev. Voodoo
    Volunteer Moderator
    Posted 3 years ago #

    I block tag and category access to robots for personal reasons

    I only want one link on google to my content, I dont want a category based, tag based, and straight permalink based going to the same place

    I don't see anything wrong with your setup

    If you are just blocking cats for 'duplicate' reasons, there has been plenty of talk on this and Google does not punish you for duplicate content because you have multiple paths to the same content

  3. Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?

    That section looks ... weird to me. I can't think of why. I WANT to say you need to go /*/feed but I could be wrong.

    Also the disallow/allow calls for the bots are not wrong, but that'd be my first stop on the fix parade. Change 'em to JUST Allow: /

    IMO, this is a good resource for a healthy robots: http://perishablepress.com/wordpress-robots-rules/

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags