Virtual Robots.txt

Description

Replaces the default virtual robots.txt generated by WordPress with an editable one, and deletes any physical robots.txt file that may already exist.

The Long Version

Virtual Robots.txt is a simple WordPress plugin for easily editing the robots.txt content on your website. Because the goal of any CMS is to be as self-contained as possible, and because WordPress itself contains a default “virtual” robots.txt file generated by the wp_options table, it makes sense to maintain a virtual robots.txt whenever possible to make site management and migration more controlled.

Instead of simply using the built-in robots_txt filter, however, this plugin creates a new hook for the main parse_query action to ensure that no other WordPress plugin (or theme) can alter the contents of robots.txt after it has been edited. Although the current version does not directly “disable” the robots_txt filter, in effect that filter is completely ignored by ending the hook sequence before robots_txt can be called.

Unlike similar plugins (or WordPress itself), it includes a default “allow all” ruleset upon activation. The theory behind this is that playing things safe is always the best course of action for inexperienced webmasters; allowing all robots to access all site content is not only recommended by Google but also makes sense in the vast majority of cases. While the form value will persist on plugin deactivation, the option value will be completely deleted in the case of the plugin being removed from your site, in an effort at clean code and to avoid any confusion or conflicts.

The plugin disables 100% the robots_txt filter because stops execution before this filter is triggered. It uses a very early hook “parse_query” that contains the context and, for a /robots.txt request, echoes the form content and ends execution of PHP.

The normal load sequence for /robots.txt requests:

1. (load plugin hooks)
2. init
3. wp
4. parse_query
5. template_redirect
6. robots_txt <- filter

When using this plugin, the sequence changes:

1. (load plugin hooks)
2. init
3. wp
4. parse_query <- detect context via WP is_robots method, show the form content and ends execution.

Compatibility

This plugin has been designed for use on LEMP (Nginx) web servers with PHP 7.0 and MySQL 5.7 to achieve best performance. All of our plugins are meant for single site WordPress installations only; for both performance and security reasons, we highly recommend against using WordPress Multisite for the vast majority of projects.

Plugin Features

  • Settings Page: Yes
  • Premium Version Available: Yes (SEO Genius)
  • Includes Media (Images, Icons, Etc): No
  • Includes CSS: No
  • Database Storage: Yes
    • Transients: No
    • Options: Yes
    • Creates New Tables: No
  • Database Queries: /robots.txt + Backend (Options API)
  • Must-Use Support: Yes (Use With Autoloader)
  • Multisite Support: No
  • Uninstalls Data: Yes

WP Admin Notices

This plugin generates multiple Admin Notices in the WP Admin dashboard. The first is a notice that fires during plugin activation which recommends several related free plugins that we believe will enhance this plugin’s features; this notice will re-appear approximately once every 6 months as our code and recommendations evolve. The second is a notice that fires a few days after plugin activation which asks for a 5-star rating of this plugin on its WordPress.org profile page. This notice will re-appear approximately once every 9 months. These notices can be dismissed by clicking the (x) symbol in the upper right of the notice box. These notices may annoy or confuse certain users, but are appreciated by the majority of our userbase, who understand that these notices support our free contributions to the WordPress community while providing valuable (free) recommendations for optimizing their website.

If you feel that these notices are too annoying, than we encourage you to consider one or more of our upcoming premium plugins that combine several free plugin features into a single control panel, or even consider developing your own plugins for WordPress, if supporting free plugin authors is too frustrating for you. A final alternative would be to place the defined constant mentioned below inside of your wp-config.php file to manually hide this plugin’s nag notices:

define('DISABLE_NAG_NOTICES', true);

Note: This defined constant will only affect the notices mentioned above, and will not affect any other notices generated by this plugin or other plugins, such as one-time notices that communicate with admin-level users.

Code Inspiration

This plugin was partially inspired either in “code or concept” by the open-source software and discussions mentioned below:

Recommended Plugins

We invite you to check out a few other related free plugins that our team has also produced that you may find especially useful:

Premium Plugins

We invite you to check out a few premium plugins that our team has also produced that you may find especially useful:

Related Services

The following paid services are available on a case-by-case basis to upstanding members of the WordPress community:

Special Thanks

We thank the following groups for their generous contributions to the WordPress community which have particularly benefited us in developing our own free plugins and paid services:

Disclaimer

We released this plugin in response to our managed hosting clients asking for better access to their server, and our primary goal will remain supporting that purpose. Although we are 100% open to fielding requests from the WordPress community, we kindly ask that you keep the above mentioned goals in mind, thanks!

Installation

  1. Upload the plugin files to /wp-content/plugins/virtual-robotstxt-littlebizzy
  2. Activate the plugin through the ‘Plugins’ screen in WordPress
  3. Check your WP Admin dashboard (admin users only) and footer area to view the statistics
  4. For Apache servers, it should work aotomatically
  5. For Nginx servers (without Apache in front) make sure try_files sends requests to your PHP processor:

    location = /robots.txt {
    try_files $uri $uri/ /index.php?$args;
    allow all;
    log_not_found off;
    access_log off;
    }

FAQ

Installation Instructions
  1. Upload the plugin files to /wp-content/plugins/virtual-robotstxt-littlebizzy
  2. Activate the plugin through the ‘Plugins’ screen in WordPress
  3. Check your WP Admin dashboard (admin users only) and footer area to view the statistics
  4. For Apache servers, it should work aotomatically
  5. For Nginx servers (without Apache in front) make sure try_files sends requests to your PHP processor:

    location = /robots.txt {
    try_files $uri $uri/ /index.php?$args;
    allow all;
    log_not_found off;
    access_log off;
    }

Does this plugin replace the need for my physical robots.txt file?

Yes. Currently its a totally virtual robots.txt solution that will “delete” any physical robots.txt that exists. The reason for this is to make WordPress as self-contained as possible, and since there is no real downside to managing robots.txt in such a virtual fashion. In future versions, we may create the option to switch between either a virtual or physical file, but for now its totally virtual as explained.

Does this plugin edit a physical robots.txt or only store a virtual robots.txt content?

This plugin will NOT edit a physical robots.txt file and only offers a virtual robots.txt content via the wp_options table.

How can I change this plugin’s settings?

Under the Settings menu on the left hand site, a new link is generated called Robots.txt

Are there any performance drawbacks to using this plugin?

Absolutely not. All wp_options are cached by default in WordPress and loaded prior to other data. In a theoretical world, a physical robots.txt file may in fact load more quickly, but all purposes both humans and robots wouldn’t notice a difference when using a virtual file. Please note that if you are using CloudFlare or other special cache systems, you may need to clear those to see your robots.txt update correctly. However, traditional caches such as page cache plugins or oboject caching (i.e. Memcached or Redis) should not effect the content of this plugin.

I’m using Nginx and this plugin is not working?

Make sure you are sending requests for the static file robots.txt via your PHP proessor such as PHP-FPM:

location = /robots.txt {
    try_files $uri $uri/ /index.php?$args;
    allow all;
    log_not_found off;
    access_log off;
}
I have a suggestion, how can I let you know?

Please avoid leaving negative reviews in order to get a feature implemented. Instead, we kindly ask that you post your feedback on the wordpress.org support forums by tagging this plugin in your post. If needed, you may also contact our homepage.

Reviews

Read all 4 reviews

Contributors & Developers

“Virtual Robots.txt” is open source software. The following people have contributed to this plugin.

Contributors

Translate “Virtual Robots.txt” into your language.

Interested in development?

Browse the code, check out the SVN repository, or subscribe to the development log by RSS.

Changelog

1.0.6

  • tested with WP 4.9
  • added support for define('DISABLE_NAG_NOTICES', true);

1.0.5

  • optimized plugin code
  • updated recommended plugins
  • added rating request

1.0.4

  • minor code tweaks
  • updated recommended plugins

1.0.3

  • added recommended plugins

1.0.2

  • tested with WP 4.8

1.0.1

  • changed menu link name
  • added tips to settings page
  • changed folder name

1.0.0

  • initial release (private)