Google bot ignores robots.txt

  • mediacity

    (@mediacity)


    14 years, 8 months ago

    Hey,

    I have a problem with google bot. It ignores my robots.txt file and crawls everything. I would like to block tag, author, category, page and some other directories. File is put in my root directory.

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /author
    Disallow: /tag
    Disallow: /page
    Disallow: /archives
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Disallow: /pogoji-uporabe/
    Disallow: /pravno-obvestilo/
    Allow: /wp-content/uploads

    User-agent: Googlebot
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.xlsx $
    Disallow: /*.doc$
    Disallow: /*.pdf$
    Disallow: /*.zip$

    Sitemap: http://www.example.com/sitemap.xml

Viewing 9 replies - 1 through 9 (of 9 total)
Viewing 9 replies - 1 through 9 (of 9 total)

The topic ‘Google bot ignores robots.txt’ is closed to new replies.