WordPress.org

1. jtroyer
   Member
   Posted 9 months ago #
   

   I'm using multisite with subdirectories, ie blog.example.com/foo and blog.example.com/bar. So I need a robots.txt file at the root blog.example.com that disallows not only Disallow: /wp- but that also Disallows /foo/wp- /bar/wp- etc.

   Do you have any suggestions or examples for this kind of setup? I'm going with the following for now:
   

   # robots.txt
   User-agent: *
   Disallow: /*?
   Disallow: /wp-
   Disallow: /feed/
   Disallow: */feed/
   Disallow: /cgi-bin/
   Disallow: /comments/
   Disallow: */comments/
   Disallow: /trackback/
   Disallow: */trackback/
   Disallow: /wp-admin/
   Disallow: /wp-content/
   Disallow: /wp-includes/
   Disallow: /wp-login.php
   Disallow: /wp-content/cache/
   Disallow: /wp-content/themes/
   Disallow: /wp-content/plugins/
   Disallow: /*/wp-admin/
   Disallow: /*/wp-content/
   Disallow: /*/wp-includes/
   Disallow: /*/wp-login.php
   Disallow: /*/wp-content/cache/
   Disallow: /*/wp-content/themes/
   Disallow: /*/wp-content/plugins/
   Allow: /

   Also, why do you include both the 'wp-' line as well as the full directories?

   http://wordpress.org/extend/plugins/multisite-robotstxt-manager/

2. jtroyer
   Member
   Posted 9 months ago #
   

   Also a nice feature would be an option to insert sitemap fields for each blog in my system when using subdirectories. (I have about 40.)

3. tribalNerd
   Member
   Plugin Author
   Posted 4 months ago #
   

   Hello jtroyer,
   I've updated the plugin to version 0.3.1, it will allow you to insert a unique sitemap url for each Website. I don't have any examples on what you're trying to do, however what you posted does look correct.

   I include both wp- and the full dir, because not all search engines read the robots.txt file the same.

   Please let me know if you have any other questions or find any issues.

   ~Chris

Reply

You must log in to post.