WordPress.org

Ready to get started?Download WordPress

Forums

Issue with robot.txt (3 posts)

  1. dasickle
    Member
    Posted 1 year ago #

    Hi All!

    So I have been trying to plug the holes on my wordpress site to get rid of duel content and have had pretty good luck with my robot.txt file it has blocked a lot of unwanted stuff BUT this morning checked my site in google and some things are still leaking out :

    domainname.com/blog/tag/blah/
    and
    domainname.com/blog/category/blah/

    Below is my robot.txt code:

    Sitemap: http://domain.com/sitemap_index.xml

    User-agent: Mediapartners-Google*
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

    # global
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /trackback/
    Disallow: /feed/
    Disallow: /comments/
    Disallow: /category/*/*
    Disallow: */trackback/
    Disallow: */feed/
    Disallow: */comments/
    Disallow: /*?
    Disallow: /go/
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: ?comments=*
    Disallow: /search?
    Disallow: /?p=*
    Allow: /wp-content/uploads/

    Do I just add Disallow: /blog/category/*/* and Disallow:/blog/tag/*

    Any ideas?

  2. Tara
    Member
    Posted 1 year ago #

  3. dasickle
    Member
    Posted 1 year ago #

    Hi t-p

    I have read it. I have gotten all of the settings from there and then added some more even. yet those two things are still not getting blocked.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags