Am I using an incorrect configuration in robots.txt

  • Yohan Perera

    (@thevirtualpreacher)


    13 years, 1 month ago

    Hi,

    I am not sure if I am using a correctly configured robots.txt in my WordPress installation.

    Google Web Master tools is reporting that the configuration prevents Google from indexing my “tags”

    I see that it’s blocking Google from indexing my categories. Is this a healthy SEO practice???

    This is my current configuration,

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: http://www.virtualpreacher.org/sitemap.xml

Viewing 2 replies - 1 through 2 (of 2 total)
  • Rev. Voodoo

    (@rvoodoo)

    13 years, 1 month ago

    I block tag and category access to robots for personal reasons

    I only want one link on google to my content, I dont want a category based, tag based, and straight permalink based going to the same place

    I don’t see anything wrong with your setup

    If you are just blocking cats for ‘duplicate’ reasons, there has been plenty of talk on this and Google does not punish you for duplicate content because you have multiple paths to the same content

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    🏳️‍🌈 Advisor and Activist

    13 years, 1 month ago

    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?

    That section looks … weird to me. I can’t think of why. I WANT to say you need to go /*/feed but I could be wrong.

    Also the disallow/allow calls for the bots are not wrong, but that’d be my first stop on the fix parade. Change ’em to JUST Allow: /

    IMO, this is a good resource for a healthy robots: http://perishablepress.com/wordpress-robots-rules/

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Am I using an incorrect configuration in robots.txt’ is closed to new replies.

Tags