WordPress.org

Ready to get started?Download WordPress

Forums

robots.txt file (2 posts)

  1. David Radovanovic
    Member
    Posted 1 year ago #

    For some reason the root of my WPMU has created a robots.txt file that has disallowed search engines with this:

    User-agent: *
    Disallow: /
    Crawl-delay: 50

    1. Don't the individual sites get there own robots.txt file or do they inherit the one above in the WPMU root?
    2. I manually changed the robots.txt file in the root to this:

    # robots.txt
    
    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    
    # Google AdSense
    User-agent: Mediapartners-Google
    Disallow:
    
    # digg mirror
    User-agent: duggmirror
    Disallow: /
    
    User-agent: *
    Disallow: /*?
    Disallow: /wp-
    Disallow: /feed/
    Disallow: */feed/
    Disallow: /cgi-bin/
    Disallow: /comments/
    Disallow: */comments/
    Disallow: /trackback/
    Disallow: */trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /wp-login.php
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-content/plugins/
    Allow: /

    Though I'm not sure I want the main site "DOMAIN_CURRENT_SITE" to be have that same robots.txt file.

    Thanks!!

  2. If it was a real robots.txt file, then it wasn't Multisite core but a plugin.

    You'll want a plugin like http://wordpress.org/extend/plugins/multisite-robotstxt-manager/ to manage it per site.

Topic Closed

This topic has been closed to new replies.

About this Topic