shawesome robots.txt – Per-Site Rules for Single Sites and Multisite

Description

WordPress generates a virtual robots.txt with the same rules for every site. shawesome robots.txt lets you replace it – on a single site, or separately for every site of a multisite network, e.g. to block a staging domain or to point each site to its own sitemap.

Edit in the admin

The plugin page is under shawesome → robots.txt (in a network in the network admin) and has three tabs: robots.txt, Settings and FAQ. On a single site, the robots.txt tab shows the editor. In a network, it lists all sites with their mode, the source of their robots.txt, the active SEO plugins and a status – sites with errors first, filterable by status and searchable – and every site has its own editor. Next to the editor, the output shows a preview of the robots.txt after saving and, on request, the robots.txt crawlers currently get. Unsaved changes are marked and can be discarded. The content is stored in the site’s database options – it survives domain changes and is removed together with the site. Leave it empty to keep the robots.txt generated by WordPress and SEO plugins.

A button inserts example rules: the WordPress default plus the sitemap of the site.

Options per site

The editor of each site also sets the mode (see below) and two groups of options. A checked option always blocks or leaves out something.

Crawlers

• a Crawl-delay (off by default, 10 seconds when switched on; Google ignores it, Bing and Yandex for example follow it)
• crawlers to block, one by one: AI crawlers (training, AI search, requests on behalf of users), SEO tools (Ahrefs, Semrush, Moz, Majestic and others), web archives, social media link previews and search engines – the last two are marked as not recommended

Paths and files

• resources and assets: CSS and JavaScript files, images and PDF files are allowed unless checked
• WordPress cleanup: internal search results, reply links of comments, feeds of posts and archives
• WooCommerce (when active): cart, checkout, my account, “add to cart” links and the folders for logs and downloadable files
• the sitemap: by default, a Sitemap line for the sitemap of WordPress, or of Yoast SEO, Rank Math or All in One SEO while their sitemap replaces it, is added, unless the robots.txt already links it; it can be left out

These lines are added at the end of the robots.txt, in both modes: blocked crawlers as a group of their own, everything else as a group for all crawlers (User-agent: *), which crawlers with a group of their own do not read. Like all of robots.txt, this only works with crawlers that follow it.

Or deploy a file

A plain text file in /wp-content/shawesome-robots-txt/ overrides the database content, e.g. for Git-based deployments. The file name is derived from the site’s domain, with dots replaced by hyphens:

• www.example.com → www-example-com.txt
• shop.example.com → shop-example-com.txt

As an unambiguous alternative, the domain with dots is accepted as well (www.example.com.txt); the hyphenated name takes precedence. While a file exists, the editor shows its content and the rules stored in the database read-only and offers to copy the file into the database; mode and options can still be changed. The switch “Ignore the override file” in the editor delivers the database rules instead, e.g. while a deployment cannot remove the file; it stays on if the file is deployed again. An empty file (or one with only whitespace) stands for the WordPress default, even if the site has rules in the database; the crawler options are still added.

Override files can be read by anyone who knows their address, like the robots.txt itself. If WordPress can write there, the plugin creates the directory with an empty index.php, so that a web server with directory listing does not show the file names (and with them the domains of a network). To keep the files out of the web root, set SHAWESOME_ROBOTS_TXT_DIR to a directory outside of it.

Which sites get their own robots.txt

Crawlers only request robots.txt from the root of a host. A site gets its own robots.txt if its address has no path:

• a single site at the root of its domain
• every site of a network with subdomains or domains
• in a network with subdirectories: the main site and sites with their own domain

Sites with a path (example.com/blog/) are served by the robots.txt of their host. By default, the plugin adds rules for them there – also in “Exact” mode, and it can be switched off:

• Disallow: /blog/ for sites hidden from search engines (WordPress only adds noindex to their pages)
• Sitemap: https://example.com/blog/wp-sitemap.xml for public sites

The Disallow lines are added as a group for all crawlers (User-agent: *). Crawlers that have a group of their own in the robots.txt do not read it.

Works with Yoast SEO and All in One SEO

Yoast SEO and All in One SEO change the robots.txt output. The mode decides how your content is combined with them. The default mode is set in the settings; each site can choose its own in its editor:

• Base (default) – the content replaces the WordPress default; Yoast SEO removes the WordPress default rules (User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php in exactly this order) and appends its own rules and sitemap, All in One SEO rebuilds the rules (comments and Sitemap lines are dropped). Duplicate Sitemap lines are removed.
• Exact – the content is delivered as is, after all SEO plugins have run. Sites without a custom robots.txt (no content in the editor and no file) keep the one from WordPress and the SEO plugins.
• Blocking – all crawlers are blocked from all paths (User-agent: * / Disallow: /), e.g. for a staging site. Rules and options of the site are kept, but not delivered.

Status

The status requests the robots.txt of each site as crawlers get it and shows whether all crawlers are blocked, how many sitemaps are linked, whether crawlers that collect content for AI training are blocked, and problems such as an unreachable robots.txt. The result is kept for a time set in the settings (24 hours by default); when the overview is opened after that time, outdated sites are checked again in the background. Each site or all sites can be checked again at any time.

Physical robots.txt

If a robots.txt file exists in the web root, the web server delivers it and WordPress is never asked. The robots.txt tab shows its content and – if WordPress may write to the web root – offers to rename it to _old-robots.txt, optionally after copying its content into the editor. The file is never deleted.

Search engine visibility

Sites set to “Discourage search engines from indexing this site” get “Blocking”: all crawlers are blocked from all paths, whatever their mode – so a staging site stays out of search engines, even if the database was copied from production. WordPress itself no longer blocks such sites in its robots.txt, and SEO plugins may even allow everything. This can be switched off in the settings.

Notes

• A UTF-8 BOM is stripped and line endings are normalized.
• In a network, the plugin is network-activated and the settings apply to all sites.
• The status is requested from the site itself (a loopback request). Some servers block such requests; the status then says so, and checking all sites stops after the first ones.
• Tested with small networks; larger networks have not been tested yet. The status check requests the robots.txt of every site, which takes longer in large networks.

For developers

• Constant SHAWESOME_ROBOTS_TXT_DIR or filter shawesome_robots_txt_dir – change the directory for override files.
• Filter shawesome_robots_txt_candidate_files – change the candidate file paths for a domain.
• Filter shawesome_robots_txt_priority – change the priority of the robots_txt filter.
• Content, mode, status, the options and the switch “Ignore the override file” per site are stored in the site options shawesome_robots_txt_content, shawesome_robots_txt_mode, shawesome_robots_txt_status, shawesome_robots_txt_rules and shawesome_robots_txt_ignore_file (not autoloaded).
• All callbacks are named functions and can be removed with remove_filter().