WordPress.org

Support

Support » How-To and Troubleshooting » Google bot ignores robots.txt

Google bot ignores robots.txt

  • mediacity
    Member

    @mediacity

    Hey,

    I have a problem with google bot. It ignores my robots.txt file and crawls everything. I would like to block tag, author, category, page and some other directories. File is put in my root directory.

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /author
    Disallow: /tag
    Disallow: /page
    Disallow: /archives
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Disallow: /pogoji-uporabe/
    Disallow: /pravno-obvestilo/
    Allow: /wp-content/uploads

    User-agent: Googlebot
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.xlsx $
    Disallow: /*.doc$
    Disallow: /*.pdf$
    Disallow: /*.zip$

    Sitemap: http://www.example.com/sitemap.xml

Viewing 9 replies - 1 through 9 (of 9 total)
Viewing 9 replies - 1 through 9 (of 9 total)
  • The topic ‘Google bot ignores robots.txt’ is closed to new replies.