WordPress.org

Forums

Multisite Robots.txt Manager
[resolved] multisite using directories (3 posts)

  1. jtroyer
    Member
    Posted 2 years ago #

    I'm using multisite with subdirectories, ie blog.example.com/foo and blog.example.com/bar. So I need a robots.txt file at the root blog.example.com that disallows not only Disallow: /wp- but that also Disallows /foo/wp- /bar/wp- etc.

    Do you have any suggestions or examples for this kind of setup? I'm going with the following for now:

    # robots.txt
    User-agent: *
    Disallow: /*?
    Disallow: /wp-
    Disallow: /feed/
    Disallow: */feed/
    Disallow: /cgi-bin/
    Disallow: /comments/
    Disallow: */comments/
    Disallow: /trackback/
    Disallow: */trackback/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /wp-login.php
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /wp-content/plugins/
    Disallow: /*/wp-admin/
    Disallow: /*/wp-content/
    Disallow: /*/wp-includes/
    Disallow: /*/wp-login.php
    Disallow: /*/wp-content/cache/
    Disallow: /*/wp-content/themes/
    Disallow: /*/wp-content/plugins/
    Allow: /

    Also, why do you include both the 'wp-' line as well as the full directories?

    http://wordpress.org/extend/plugins/multisite-robotstxt-manager/

  2. jtroyer
    Member
    Posted 2 years ago #

    Also a nice feature would be an option to insert sitemap fields for each blog in my system when using subdirectories. (I have about 40.)

  3. tribalNerd
    Member
    Plugin Author

    Posted 2 years ago #

    Hello jtroyer,
    I've updated the plugin to version 0.3.1, it will allow you to insert a unique sitemap url for each Website. I don't have any examples on what you're trying to do, however what you posted does look correct.

    I include both wp- and the full dir, because not all search engines read the robots.txt file the same.

    Please let me know if you have any other questions or find any issues.

    ~Chris

Topic Closed

This topic has been closed to new replies.

About this Plugin

  • Multisite Robots.txt Manager
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic