View the Install Guide |
Understanding the Default Settings
- When you first enter the plugin admin via the Network Admin, the shown robots.txt file is the default "network only" or "network wide" working copy. Modify the default robots.txt file, save the default file, and when ready click the "publish to network" button to duplicate the robots.txt file to all Network Websites.
Create / Manage
- The Defaults tab contains an inactive, "network only" or "network wide" working copy of the robots.txt file. Modify the default robots.txt file, save the default file, and when ready click the "publish to network" button to duplicate the robots.txt file to all Network Websites.
Sitemap URLs and Structure
The Sitemap URL Structure feature uses 3 different [shortcodes] to customize how sitemap urls are rendered on a robots.txt file. The shortcodes get a websites url and then break the url apart, separating the domain name from the domain extension.
Example Structure: http://WEBSITE_URL/sitemap.xml
The [bracket] within the url automatically gets replaced by the plugin (You Will Use The Brackets).
The default sitemap structure within the network admin, "must" use the [bracket] structure, otherwise when you publish to the network, the sitemap urls will not render correctly. You can directly modify a Websites sitemap structure or enter a full sitemap url, by selected the site from the drop down menu above, then click the "change sites" button.
Sitemap URL Structure
Robots.txt Files within Directories
This plugin WILL render robots.txt files within directories - however,
Search Engine Spiders only read robots.txt files found within the root directory of a Website. Spiders do not read robots.txt files within directories, such as: domain.com/PATH-or-FOLDER/robots.txt is NOT a valid location.
From Google: "The robots.txt file must be in the top-level directory of the host.....Crawlers will not check for robots.txt files in sub-directories." [ source ]
Testing Robots.txt Files
- Use Google's Webmaster Tools to Validate your Robots.txt Files.... with Google at least.:
Log into your Google Account and access the Log into your Webmaster Tools feature. Select a Website or Add a Website....
On the Webmaster Tools Home page, click the site you want.
- Under Health, click Blocked URLs.
- If it is not already selected, click the Test robots.txt tab.
- Copy the content of your robots.txt file, and paste it into the first box.
- In the URLs box, list the site to test against.
- In the User-agents list, select the user-agents you want.
New Website Added to Network
- If all Websites use the saved Network default robots.txt file, click the "publish to network" button to copy the default robots.txt file over to any new Websites you have.
- Per Site: Change to the Website in the dropdown. Then click the "reset this website" button to copy the default robots.txt file to this Website. If needed, modify the robots.txt file and click the "update this website" button once done.
- Disable a Website: To disable the MS Robots.txt Manager on a Website, select the Website from the dropdown menu, then click the "change sites" button. With the Website's robots.txt file open, click the "disable this website" button. This will clear the robots.txt file and sitemap structure settings for this Website only, making the WordPress default robots.txt file display.
- Disable across the Network: Select the default robots.txt file within the Text Area, click the delete on your keyboard, then click the "publish to network" button. You can not save a blank default robots.txt file, but you can publish a blank robots.txt file, which will disable the robots.txt file option for each Website within the Network.
- Reset Default: Something wrong? No worries! When viewing the Networks robots.txt file, click the "reset to default" button to replace the displayed robots.txt file with the core "coded in" default robots.txt file.
- Reset Website: To reset a Websites robots.txt file, change to the Website within the dropdown, then click the "reset this website" button to pull in the "Networks Default Robots.txt file" (not the coded in default file).
Presets / Examples Tab
- This feature allows you to quickly duplicate premade robots.txt files and a sitemap structure url, to either the default network wide robots.txt file or a selected Websites robots.txt file.
To use: Select the Network or a Website from the dropdown. Check the box to add a sitemap structure, modify/enter a Sitemap Structure (not required). Finally, click the "set as default" button above the robots.txt file example you want to use.
Presets can also use the Sitemap URL Structure setting. Read above on how to use this feature.
Recommended Sitemap Plugins
For "real" Multisite HOST Networks, use the WordPress plugin: BWP Google XML Sitemaps - This plugin will list each Websites Sitemap URL's in the Root Network Website's robots.txt file.