A Multisite Robots.txt Manager - Quickly and easily manage all robots.txt files on a WordPress Multisite Website Network.
The Sitemap URL Structure feature uses 3 different [shortcodes] to customize how sitemap urls are rendered on a robots.txt file. The shortcodes get a websites url and then break the url apart, separating the domain name from the domain extension.
Example Structure: http://WEBSITE_URL/sitemap.xml
The [bracket] within the url automatically gets replaced by the plugin (You Will Use The Brackets).
The default sitemap structure within the network admin, "must" use the [bracket] structure, otherwise when you publish to the network, the sitemap urls will not render correctly. You can directly modify a Websites sitemap structure or enter a full sitemap url, by selected the site from the drop down menu above, then click the "change sites" button.
Example sitemap structures you can use.
[EXT] = .com/net, etc.
Always include the http:// with the Sitemap URL Structure.
This plugin WILL render robots.txt files within directories - however,
Search Engine Spiders only read robots.txt files found within the root directory of a Website. Spiders do not read robots.txt files within directories, such as: domain.com/PATH-or-FOLDER/robots.txt is NOT a valid location.
From Google: "The robots.txt file must be in the top-level directory of the host.....Crawlers will not check for robots.txt files in sub-directories." [ source ]
Log into your Google Account and access the Log into your Webmaster Tools feature. Select a Website or Add a Website....
On the Webmaster Tools Home page, click the site you want.
To use: Select the Network or a Website from the dropdown. Check the box to add a sitemap structure, modify/enter a Sitemap Structure (not required). Finally, click the "set as default" button above the robots.txt file example you want to use.
Presets can also use the Sitemap URL Structure setting. Read above on how to use this feature.
For "real" Multisite HOST Networks, use the WordPress plugin: BWP Google XML Sitemaps - This plugin will list each Websites Sitemap URL's in the Root Network Website's robots.txt file.
Requires: 3.3 or higher
Compatible up to: 3.5.2
Last Updated: 2013-4-17
Active Installs: 200+
Got something to say? Need help?