WordPress.org

Support

Support » Multisite » robots.txt file

robots.txt file

  • For some reason the root of my WPMU has created a robots.txt file that has disallowed search engines with this:

    User-agent: *
    Disallow: /
    Crawl-delay: 50

    1. Don’t the individual sites get there own robots.txt file or do they inherit the one above in the WPMU root?
    2. I manually changed the robots.txt file in the root to this:

    # robots.txt
    
    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    
    # Google AdSense
    User-agent: Mediapartners-Google
    Disallow:
    
    # digg mirror
    User-agent: duggmirror
    Disallow: /
    
    User-agent: *
    Disallow: /*?
    Disallow: /wp-
    Disallow: /feed/
    Disallow: */feed/
    Disallow: /cgi-bin/
    Disallow: /comments/
    Disallow: */comments/
    Disallow: /trackback/
    Disallow: */trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /wp-login.php
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-content/plugins/
    Allow: /

    Though I’m not sure I want the main site “DOMAIN_CURRENT_SITE” to be have that same robots.txt file.

    Thanks!!

Viewing 1 replies (of 1 total)
Viewing 1 replies (of 1 total)
  • The topic ‘robots.txt file’ is closed to new replies.