For some reason the root of my WPMU has created a robots.txt file that has disallowed search engines with this:
User-agent: * Disallow: / Crawl-delay: 50
1. Don’t the individual sites get there own robots.txt file or do they inherit the one above in the WPMU root?
2. I manually changed the robots.txt file in the root to this:
# robots.txt # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google Disallow: # digg mirror User-agent: duggmirror Disallow: / User-agent: * Disallow: /*? Disallow: /wp- Disallow: /feed/ Disallow: */feed/ Disallow: /cgi-bin/ Disallow: /comments/ Disallow: */comments/ Disallow: /trackback/ Disallow: */trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /wp-login.php Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /wp-content/plugins/ Allow: /
Though I’m not sure I want the main site “DOMAIN_CURRENT_SITE” to be have that same robots.txt file.
- The topic ‘robots.txt file’ is closed to new replies.