WordPress generates a virtual robots.txt with the same rules for every site. shawesome robots.txt lets you replace it \u2013 on a single site, or separately for every site of a multisite network, e.g. to block a staging domain or to point each site to its own sitemap.<\/p>\n\n
Edit in the admin<\/strong><\/p>\n\n The plugin page is under shawesome \u2192 robots.txt<\/em> (in a network in the network admin) and has three tabs: robots.txt<\/em>, Settings<\/em> and FAQ<\/em>. On a single site, the robots.txt tab shows the editor. In a network, it lists all sites with their mode, the source of their robots.txt, the active SEO plugins and a status \u2013 sites with errors first, filterable by status and searchable \u2013 and every site has its own editor. Next to the editor, the output shows a preview of the robots.txt after saving and, on request, the robots.txt crawlers currently get. Unsaved changes are marked and can be discarded. The content is stored in the site's database options \u2013 it survives domain changes and is removed together with the site. Leave it empty to keep the robots.txt generated by WordPress and SEO plugins.<\/p>\n\n A button inserts example rules: the WordPress default plus the sitemap of the site.<\/p>\n\n Options per site<\/strong><\/p>\n\n The editor of each site also sets the mode (see below) and two groups of options. A checked option always blocks or leaves out something.<\/p>\n\n Crawlers<\/em><\/p>\n\n Paths and files<\/em><\/p>\n\n These lines are added at the end of the robots.txt, in both modes: blocked crawlers as a group of their own, everything else as a group for all crawlers ( Or deploy a file<\/strong><\/p>\n\n A plain text file in As an unambiguous alternative, the domain with dots is accepted as well ( Override files can be read by anyone who knows their address, like the robots.txt itself. If WordPress can write there, the plugin creates the directory with an empty Which sites get their own robots.txt<\/strong><\/p>\n\n Crawlers only request robots.txt from the root of a host. A site gets its own robots.txt if its address has no path:<\/p>\n\n Sites with a path ( The Disallow lines are added as a group for all crawlers ( Works with Yoast SEO and All in One SEO<\/strong><\/p>\n\n Yoast SEO and All in One SEO change the robots.txt output. The mode decides how your content is combined with them. The default mode is set in the settings; each site can choose its own in its editor:<\/p>\n\n Status<\/strong><\/p>\n\n The status requests the robots.txt of each site as crawlers get it and shows whether all crawlers are blocked, how many sitemaps are linked, whether crawlers that collect content for AI training are blocked, and problems such as an unreachable robots.txt. The result is kept for a time set in the settings (24 hours by default); when the overview is opened after that time, outdated sites are checked again in the background. Each site or all sites can be checked again at any time.<\/p>\n\n Physical robots.txt<\/strong><\/p>\n\n If a Search engine visibility<\/strong><\/p>\n\n Sites set to \"Discourage search engines from indexing this site\" get \"Blocking\": all crawlers are blocked from all paths, whatever their mode \u2013 so a staging site stays out of search engines, even if the database was copied from production. WordPress itself no longer blocks such sites in its robots.txt, and SEO plugins may even allow everything. This can be switched off in the settings.<\/p>\n\n Notes<\/strong><\/p>\n\n For developers<\/strong><\/p>\n\n Check on the robots.txt tab that there is no physical A file in Select the mode \"Exact\" in the settings. The content is then delivered after all SEO plugins have run.<\/p><\/dd>\n It requests the robots.txt of the site as crawlers get it and shows whether all crawlers are blocked, how many sitemaps are linked and whether crawlers that collect content for AI training are blocked. The result is kept for the time set in the settings and can be checked again at any time.<\/p><\/dd>\n If the address of the site contains a path ( The settings and the content stored in the database are deleted. Files in \n
\n
User-agent: *<\/code>), which crawlers with a group of their own do not read. Like all of robots.txt, this only works with crawlers that follow it.<\/p>\n\n\/wp-content\/shawesome-robots-txt\/<\/code> overrides the database content, e.g. for Git-based deployments. The file name is derived from the site's domain, with dots replaced by hyphens:<\/p>\n\n\n
www.example.com<\/code> \u2192 www-example-com.txt<\/code><\/li>\nshop.example.com<\/code> \u2192 shop-example-com.txt<\/code><\/li>\n<\/ul>\n\nwww.example.com.txt<\/code>); the hyphenated name takes precedence. While a file exists, the editor shows its content and the rules stored in the database read-only and offers to copy the file into the database; mode and options can still be changed. The switch \"Ignore the override file\" in the editor delivers the database rules instead, e.g. while a deployment cannot remove the file; it stays on if the file is deployed again. An empty file (or one with only whitespace) stands for the WordPress default, even if the site has rules in the database; the crawler options are still added.<\/p>\n\nindex.php<\/code>, so that a web server with directory listing does not show the file names (and with them the domains of a network). To keep the files out of the web root, set SHAWESOME_ROBOTS_TXT_DIR<\/code> to a directory outside of it.<\/p>\n\n\n
example.com\/blog\/<\/code>) are served by the robots.txt of their host. By default, the plugin adds rules for them there \u2013 also in \"Exact\" mode, and it can be switched off:<\/p>\n\n\n
Disallow: \/blog\/<\/code> for sites hidden from search engines (WordPress only adds noindex<\/code> to their pages)<\/li>\nSitemap: https:\/\/example.com\/blog\/wp-sitemap.xml<\/code> for public sites<\/li>\n<\/ul>\n\nUser-agent: *<\/code>). Crawlers that have a group of their own in the robots.txt do not read it.<\/p>\n\n\n
User-agent: *<\/code>, Disallow: \/wp-admin\/<\/code>, Allow: \/wp-admin\/admin-ajax.php<\/code> in exactly this order) and appends its own rules and sitemap, All in One SEO rebuilds the rules (comments and Sitemap lines are dropped). Duplicate Sitemap lines are removed.<\/li>\nUser-agent: *<\/code> \/ Disallow: \/<\/code>), e.g. for a staging site. Rules and options of the site are kept, but not delivered.<\/li>\n<\/ul>\n\nrobots.txt<\/code> file exists in the web root, the web server delivers it and WordPress is never asked. The robots.txt tab shows its content and \u2013 if WordPress may write to the web root \u2013 offers to rename it to _old-robots.txt<\/code>, optionally after copying its content into the editor. The file is never deleted.<\/p>\n\n\n
\n
SHAWESOME_ROBOTS_TXT_DIR<\/code> or filter shawesome_robots_txt_dir<\/code> \u2013 change the directory for override files.<\/li>\nshawesome_robots_txt_candidate_files<\/code> \u2013 change the candidate file paths for a domain.<\/li>\nshawesome_robots_txt_priority<\/code> \u2013 change the priority of the robots_txt<\/code> filter.<\/li>\nshawesome_robots_txt_content<\/code>, shawesome_robots_txt_mode<\/code>, shawesome_robots_txt_status<\/code>, shawesome_robots_txt_rules<\/code> and shawesome_robots_txt_ignore_file<\/code> (not autoloaded).<\/li>\nremove_filter()<\/code>.<\/li>\n<\/ul>\n\n\n\n
shawesome-robots-txt<\/code> to \/wp-content\/plugins\/<\/code> or install the plugin via Plugins \u2192 Add New<\/em>.<\/li>\n\/wp-content\/shawesome-robots-txt\/<\/code>, e.g. www-example-com.txt<\/code>.<\/li>\nhttps:\/\/www.example.com\/robots.txt<\/code>.<\/li>\n<\/ol>\n\n\n\n
My robots.txt is not delivered.<\/h3><\/dt>\n
robots.txt<\/code> in the web root and that the site is not set to \"Discourage search engines\". In a network, the overview shows the source and the status of every site. For files, check that they are readable by the web server.<\/p><\/dd>\nI changed the content in the editor, but the old one is still delivered.<\/h3><\/dt>\n
\/wp-content\/shawesome-robots-txt\/<\/code> overrides the database content. Delete the file, copy it into the database first with the button in the editor, or switch on \"Ignore the override file\" in the editor.<\/p><\/dd>\nYoast SEO or All in One SEO changes my robots.txt.<\/h3><\/dt>\n
What does the status check?<\/h3><\/dt>\n
Does it work if WordPress is installed in a subdirectory?<\/h3><\/dt>\n
example.com\/blog\/<\/code>), crawlers request the robots.txt from example.com\/robots.txt<\/code>, which WordPress does not deliver. The plugin then does nothing and says so. If only the WordPress files are in a subdirectory and the address is the root of the domain, it works.<\/p><\/dd>\nWhat happens on uninstall?<\/h3><\/dt>\n
\/wp-content\/shawesome-robots-txt\/<\/code> and a renamed _old-robots.txt<\/code> are kept.<\/p><\/dd>\n\n<\/dl>\n\n\n1.0.0<\/h4>\n\n
\n