WordPress.org

Support

Support » How-To and Troubleshooting » Issue with robot.txt

Issue with robot.txt

  • Hi All!

    So I have been trying to plug the holes on my wordpress site to get rid of duel content and have had pretty good luck with my robot.txt file it has blocked a lot of unwanted stuff BUT this morning checked my site in google and some things are still leaking out :

    domainname.com/blog/tag/blah/
    and
    domainname.com/blog/category/blah/

    Below is my robot.txt code:

    Sitemap: http://domain.com/sitemap_index.xml

    User-agent: Mediapartners-Google*
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

    # global
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /trackback/
    Disallow: /feed/
    Disallow: /comments/
    Disallow: /category/*/*
    Disallow: */trackback/
    Disallow: */feed/
    Disallow: */comments/
    Disallow: /*?
    Disallow: /go/
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: ?comments=*
    Disallow: /search?
    Disallow: /?p=*
    Allow: /wp-content/uploads/

    Do I just add Disallow: /blog/category/*/* and Disallow:/blog/tag/*

    Any ideas?

Viewing 2 replies - 1 through 2 (of 2 total)
Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Issue with robot.txt’ is closed to new replies.