Title: shawesome robots.txt – Per-Site Rules for Single Sites and Multisite Author: shawesome Published: November 9, 2026 Last modified: November 9, 2026 --- Search plugins ![](https://ps.w.org/shawesome-robots-txt/assets/banner-772x250.png?rev=3793373) ![](https://ps.w.org/shawesome-robots-txt/assets/icon-256x256.png?rev=3793373) # shawesome robots.txt – Per-Site Rules for Single Sites and Multisite By [shawesome](https://profiles.wordpress.org/shawesome/) [Download](https://downloads.wordpress.org/plugin/shawesome-robots-txt.1.0.0.zip) * [Details](https://wordpress.org/plugins/shawesome-robots-txt/#description) * [Reviews](https://wordpress.org/plugins/shawesome-robots-txt/#reviews) * [Installation](https://wordpress.org/plugins/shawesome-robots-txt/#installation) * [Development](https://wordpress.org/plugins/shawesome-robots-txt/#developers) [Support](https://wordpress.org/support/plugin/shawesome-robots-txt/) ## Description WordPress generates a virtual robots.txt with the same rules for every site. shawesome robots.txt lets you replace it – on a single site, or separately for every site of a multisite network, e.g. to block a staging domain or to point each site to its own sitemap. **Edit in the admin** The plugin page is under _shawesome robots.txt_ (in a network in the network admin) and has three tabs: _robots.txt_, _Settings_ and _FAQ_. On a single site, the robots. txt tab shows the editor. In a network, it lists all sites with their mode, the source of their robots.txt, the active SEO plugins and a status – sites with errors first, filterable by status and searchable – and every site has its own editor. Next to the editor, the output shows a preview of the robots.txt after saving and, on request, the robots.txt crawlers currently get. Unsaved changes are marked and can be discarded. The content is stored in the site’s database options – it survives domain changes and is removed together with the site. Leave it empty to keep the robots.txt generated by WordPress and SEO plugins. A button inserts example rules: the WordPress default plus the sitemap of the site. **Options per site** The editor of each site also sets the mode (see below) and two groups of options. A checked option always blocks or leaves out something. _Crawlers_ * a Crawl-delay (off by default, 10 seconds when switched on; Google ignores it, Bing and Yandex for example follow it) * crawlers to block, one by one: AI crawlers (training, AI search, requests on behalf of users), SEO tools (Ahrefs, Semrush, Moz, Majestic and others), web archives, social media link previews and search engines – the last two are marked as not recommended _Paths and files_ * resources and assets: CSS and JavaScript files, images and PDF files are allowed unless checked * WordPress cleanup: internal search results, reply links of comments, feeds of posts and archives * WooCommerce (when active): cart, checkout, my account, “add to cart” links and the folders for logs and downloadable files * the sitemap: by default, a Sitemap line for the sitemap of WordPress, or of Yoast SEO, Rank Math or All in One SEO while their sitemap replaces it, is added, unless the robots.txt already links it; it can be left out These lines are added at the end of the robots.txt, in both modes: blocked crawlers as a group of their own, everything else as a group for all crawlers (`User-agent:*`), which crawlers with a group of their own do not read. Like all of robots.txt, this only works with crawlers that follow it. **Or deploy a file** A plain text file in `/wp-content/shawesome-robots-txt/` overrides the database content, e.g. for Git-based deployments. The file name is derived from the site’s domain, with dots replaced by hyphens: * `www.example.com` `www-example-com.txt` * `shop.example.com` `shop-example-com.txt` As an unambiguous alternative, the domain with dots is accepted as well (`www.example. com.txt`); the hyphenated name takes precedence. While a file exists, the editor shows its content and the rules stored in the database read-only and offers to copy the file into the database; mode and options can still be changed. The switch “Ignore the override file” in the editor delivers the database rules instead, e.g. while a deployment cannot remove the file; it stays on if the file is deployed again. An empty file (or one with only whitespace) stands for the WordPress default, even if the site has rules in the database; the crawler options are still added. Override files can be read by anyone who knows their address, like the robots.txt itself. If WordPress can write there, the plugin creates the directory with an empty` index.php`, so that a web server with directory listing does not show the file names( and with them the domains of a network). To keep the files out of the web root, set `SHAWESOME_ROBOTS_TXT_DIR` to a directory outside of it. **Which sites get their own robots.txt** Crawlers only request robots.txt from the root of a host. A site gets its own robots. txt if its address has no path: * a single site at the root of its domain * every site of a network with subdomains or domains * in a network with subdirectories: the main site and sites with their own domain Sites with a path (`example.com/blog/`) are served by the robots.txt of their host. By default, the plugin adds rules for them there – also in “Exact” mode, and it can be switched off: * `Disallow: /blog/` for sites hidden from search engines (WordPress only adds ` noindex` to their pages) * `Sitemap: https://example.com/blog/wp-sitemap.xml` for public sites The Disallow lines are added as a group for all crawlers (`User-agent: *`). Crawlers that have a group of their own in the robots.txt do not read it. **Works with Yoast SEO and All in One SEO** Yoast SEO and All in One SEO change the robots.txt output. The mode decides how your content is combined with them. The default mode is set in the settings; each site can choose its own in its editor: * **Base** (default) – the content replaces the WordPress default; Yoast SEO removes the WordPress default rules (`User-agent: *`, `Disallow: /wp-admin/`, `Allow:/ wp-admin/admin-ajax.php` in exactly this order) and appends its own rules and sitemap, All in One SEO rebuilds the rules (comments and Sitemap lines are dropped). Duplicate Sitemap lines are removed. * **Exact** – the content is delivered as is, after all SEO plugins have run. Sites without a custom robots.txt (no content in the editor and no file) keep the one from WordPress and the SEO plugins. * **Blocking** – all crawlers are blocked from all paths (`User-agent: *` / `Disallow:/`), e.g. for a staging site. Rules and options of the site are kept, but not delivered. **Status** The status requests the robots.txt of each site as crawlers get it and shows whether all crawlers are blocked, how many sitemaps are linked, whether crawlers that collect content for AI training are blocked, and problems such as an unreachable robots. txt. The result is kept for a time set in the settings (24 hours by default); when the overview is opened after that time, outdated sites are checked again in the background. Each site or all sites can be checked again at any time. **Physical robots.txt** If a `robots.txt` file exists in the web root, the web server delivers it and WordPress is never asked. The robots.txt tab shows its content and – if WordPress may write to the web root – offers to rename it to `_old-robots.txt`, optionally after copying its content into the editor. The file is never deleted. **Search engine visibility** Sites set to “Discourage search engines from indexing this site” get “Blocking”: all crawlers are blocked from all paths, whatever their mode – so a staging site stays out of search engines, even if the database was copied from production. WordPress itself no longer blocks such sites in its robots.txt, and SEO plugins may even allow everything. This can be switched off in the settings. **Notes** * A UTF-8 BOM is stripped and line endings are normalized. * In a network, the plugin is network-activated and the settings apply to all sites. * The status is requested from the site itself (a loopback request). Some servers block such requests; the status then says so, and checking all sites stops after the first ones. * Tested with small networks; larger networks have not been tested yet. The status check requests the robots.txt of every site, which takes longer in large networks. **For developers** * Constant `SHAWESOME_ROBOTS_TXT_DIR` or filter `shawesome_robots_txt_dir` – change the directory for override files. * Filter `shawesome_robots_txt_candidate_files` – change the candidate file paths for a domain. * Filter `shawesome_robots_txt_priority` – change the priority of the `robots_txt` filter. * Content, mode, status, the options and the switch “Ignore the override file” per site are stored in the site options `shawesome_robots_txt_content`, `shawesome_robots_txt_mode`,` shawesome_robots_txt_status`, `shawesome_robots_txt_rules` and `shawesome_robots_txt_ignore_file`( not autoloaded). * All callbacks are named functions and can be removed with `remove_filter()`. ## Screenshots [⌊Network settings: default mode, blocking of hidden sites, rules for sites with a path and the status interval.⌉⌊Network settings: default mode, blocking of hidden sites, rules for sites with a path and the status interval.⌉[ Network settings: default mode, blocking of hidden sites, rules for sites with a path and the status interval. [⌊Network with subdomains: every site with its mode, the source of its robots.txt and its status.⌉⌊Network with subdomains: every site with its mode, the source of its robots.txt and its status.⌉[ Network with subdomains: every site with its mode, the source of its robots.txt and its status. [⌊Network with subdirectories: sites with a path are served by the robots.txt of their host.⌉⌊Network with subdirectories: sites with a path are served by the robots. txt of their host.⌉[ Network with subdirectories: sites with a path are served by the robots.txt of their host. [⌊Editor of a site: mode, rules and the output after saving.⌉⌊Editor of a site: mode, rules and the output after saving.⌉[ Editor of a site: mode, rules and the output after saving. [⌊Crawler options: block AI crawlers, SEO tools, web archives, social media and search engines one by one.⌉⌊Crawler options: block AI crawlers, SEO tools, web archives, social media and search engines one by one.⌉[ Crawler options: block AI crawlers, SEO tools, web archives, social media and search engines one by one. [⌊Path and file options: resources, WordPress cleanup and the sitemap.⌉⌊Path and file options: resources, WordPress cleanup and the sitemap.⌉[ Path and file options: resources, WordPress cleanup and the sitemap. [⌊WooCommerce options: cart, checkout, my account, "add to cart" links and the folders for logs and files.⌉⌊WooCommerce options: cart, checkout, my account, "add to cart" links and the folders for logs and files.⌉[ WooCommerce options: cart, checkout, my account, “add to cart” links and the folders for logs and files. [⌊Single site: the editor with the robots.txt generated by WordPress.⌉⌊Single site: the editor with the robots.txt generated by WordPress.⌉[ Single site: the editor with the robots.txt generated by WordPress. ## Installation 1. Upload the folder `shawesome-robots-txt` to `/wp-content/plugins/` or install the plugin via _Plugins Add New_. 2. Activate the plugin – in a network, network-activate it. 3. Edit the robots.txt under _shawesome robots.txt_ (single site) or _Network Admin shawesome robots.txt_ (network) – or place a text file per domain in `/wp-content/ shawesome-robots-txt/`, e.g. `www-example-com.txt`. 4. Check the result with the “Open live robots.txt” link or at `https://www.example. com/robots.txt`. ## FAQ ### My robots.txt is not delivered. Check on the robots.txt tab that there is no physical `robots.txt` in the web root and that the site is not set to “Discourage search engines”. In a network, the overview shows the source and the status of every site. For files, check that they are readable by the web server. ### I changed the content in the editor, but the old one is still delivered. A file in `/wp-content/shawesome-robots-txt/` overrides the database content. Delete the file, copy it into the database first with the button in the editor, or switch on “Ignore the override file” in the editor. ### Yoast SEO or All in One SEO changes my robots.txt. Select the mode “Exact” in the settings. The content is then delivered after all SEO plugins have run. ### What does the status check? It requests the robots.txt of the site as crawlers get it and shows whether all crawlers are blocked, how many sitemaps are linked and whether crawlers that collect content for AI training are blocked. The result is kept for the time set in the settings and can be checked again at any time. ### Does it work if WordPress is installed in a subdirectory? If the address of the site contains a path (`example.com/blog/`), crawlers request the robots.txt from `example.com/robots.txt`, which WordPress does not deliver. The plugin then does nothing and says so. If only the WordPress files are in a subdirectory and the address is the root of the domain, it works. ### What happens on uninstall? The settings and the content stored in the database are deleted. Files in `/wp-content/ shawesome-robots-txt/` and a renamed `_old-robots.txt` are kept. ## Reviews There are no reviews for this plugin. ## Contributors & Developers “shawesome robots.txt – Per-Site Rules for Single Sites and Multisite” is open source software. The following people have contributed to this plugin. Contributors * [ shawesome ](https://profiles.wordpress.org/shawesome/) [Translate “shawesome robots.txt – Per-Site Rules for Single Sites and Multisite” into your language.](https://translate.wordpress.org/projects/wp-plugins/shawesome-robots-txt) ### Interested in development? [Browse the code](https://plugins.trac.wordpress.org/browser/shawesome-robots-txt/), check out the [SVN repository](https://plugins.svn.wordpress.org/shawesome-robots-txt/), or subscribe to the [development log](https://plugins.trac.wordpress.org/log/shawesome-robots-txt/) by [RSS](https://plugins.trac.wordpress.org/log/shawesome-robots-txt/?limit=100&mode=stop_on_copy&format=rss). ## Changelog #### 1.0.0 * Initial release. ## Meta * Version **1.0.0** * Last updated **1 day ago** * Active installations **Fewer than 10** * WordPress version ** 6.4 or higher ** * Tested up to **7.1.4** * PHP version ** 7.4 or higher ** * Tags * [crawler](https://wordpress.org/plugins/tags/crawler/)[multisite](https://wordpress.org/plugins/tags/multisite/) [network](https://wordpress.org/plugins/tags/network/)[robots.txt](https://wordpress.org/plugins/tags/robots-txt/) [seo](https://wordpress.org/plugins/tags/seo/) * [Advanced View](https://wordpress.org/plugins/shawesome-robots-txt/advanced/) ## Ratings No reviews have been submitted yet. [Your review](https://wordpress.org/support/plugin/shawesome-robots-txt/reviews/#new-post) [See all reviews](https://wordpress.org/support/plugin/shawesome-robots-txt/reviews/) ## Contributors * [ shawesome ](https://profiles.wordpress.org/shawesome/) ## Support Got something to say? Need help? [View support forum](https://wordpress.org/support/plugin/shawesome-robots-txt/)