This plugin hasn’t been updated in over 2 years. It may no longer be maintained or supported and may have compatibility issues when used with more recent versions of WordPress.

KB Robots.txt

Description

When robots (like the Googlebot) crawl your site, they begin by requesting http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within WordPress (using Options -> Robots.txt).

Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the WordPress admin menu.

This plugin should work with most versions of WordPress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.

Limitations

Note that robots make only top-level requests for robots.txt files. If you have WordPress installed in a subdomain (e.g. http://blog.example.com/) or in your root (e.g. http://example.com/), this plugin will work as intended. But if you have WordPress installed in a subdirectory (e.g. http://example.com/blog/), then this plugin won’t do much for you, since the search engines won’t look for http://example.com/blog/robots.txt, only for http://example.com/robots.txt.

Also, requires that you be using some form of permalink structure. If links to blog posts look like http://example.com/index.php?p=36, this won’t work.

Support

If you post your support questions as comments below, I probably won’t see them. If the FAQs don’t answer your questions, you can post support questions at the KB Robots.txt plugin page on my site.

License

This plugin is provided “as is” and without any warranty or expectation of function. I’ll probably try to help you if you ask nicely, but I can’t promise anything. You are welcome to use this plugin and modify it however you want, as long as you give credit where it is due.

I give this away for free. If you run a blog farm (WP-MU), you can’t charge people to use this plugin. Unless, of course, you want to share the wealth with the person who coded this…

Installation

  1. Upload kb_robots-txt.php to the /wp-content/plugins/ directory.
  2. Activate the plugin through the ‘Plugins’ menu in WordPress.
  3. Go to the new ‘Options => KB Robots.txt’ admin page. In the box, write whatever you want your robots.txt file to say. There are examples below the box.
  4. Test that it worked by going to http://yourblog.example.com/robots.txt to see your new robots.txt file.

FAQ

Do I need a robots.txt file?

All well-behaved robots (which includes all search engine spiders) will request http://example.com/robots.txt before doing anything else. If you wish to allow all robots access to your entire site, then you do not need this plugin.

You need robots.txt (and therefore this plugin) to achieve any of the following goals:

  • Ban a specific robot (e.g. don’t allow MSN Search to crawl your site)
  • Ban all but one robot (allow only Google to index your site)
  • Ban robots from indexing a particular part of your site (e.g. preventing indexing of feeds and trackback URLs)
  • Ban robots from indexing any of your site

You can find much more information in the robotstxt.org FAQ. You might also want to peruse some tips for optimizing your blog’s rank in Google (though I do not necessarily endorse these tips).

I have a question that isn’t addressed here.

You may ask questions by posting a comment to the KB Robots.txt plugin page. (If you post your support questions as comments below, I probably won’t see them.)

Reviews

Отлично!

Отличная штука! Помогла мне вернуть Robots.txt, у кого нет роботса из за sitemap. это виртуальный редактор.

Read all 1 review

Contributors & Developers

“KB Robots.txt” is open source software. The following people have contributed to this plugin.

Contributors