When robots (like the Googlebot) crawl your site, they begin by requesting
http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within WordPress (using
Options -> Robots.txt).
Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the WordPress admin menu.
This plugin should work with most versions of WordPress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.
Note that robots make only top-level requests for robots.txt files. If you have WordPress installed in a subdomain (e.g.
http://blog.example.com/) or in your root (e.g.
http://example.com/), this plugin will work as intended. But if you have WordPress installed in a subdirectory (e.g.
http://example.com/blog/), then this plugin won’t do much for you, since the search engines won’t look for
http://example.com/blog/robots.txt, only for
Also, requires that you be using some form of permalink structure. If links to blog posts look like
http://example.com/index.php?p=36, this won’t work.
If you post your support questions as comments below, I probably won’t see them. If the FAQs don’t answer your questions, you can post support questions at the KB Robots.txt plugin page on my site.
This plugin is provided “as is” and without any warranty or expectation of function. I’ll probably try to help you if you ask nicely, but I can’t promise anything. You are welcome to use this plugin and modify it however you want, as long as you give credit where it is due.
I give this away for free. If you run a blog farm (WP-MU), you can’t charge people to use this plugin. Unless, of course, you want to share the wealth with the person who coded this…
- Activate the plugin through the ‘Plugins’ menu in WordPress.
- Go to the new ‘Options => KB Robots.txt’ admin page. In the box, write whatever you want your robots.txt file to say. There are examples below the box.
- Test that it worked by going to
http://yourblog.example.com/robots.txtto see your new robots.txt file.
- Do I need a robots.txt file?
All well-behaved robots (which includes all search engine spiders) will request http://example.com/robots.txt before doing anything else. If you wish to allow all robots access to your entire site, then you do not need this plugin.
You need robots.txt (and therefore this plugin) to achieve any of the following goals:
- Ban a specific robot (e.g. don’t allow MSN Search to crawl your site)
- Ban all but one robot (allow only Google to index your site)
- Ban robots from indexing a particular part of your site (e.g. preventing indexing of feeds and trackback URLs)
- Ban robots from indexing any of your site
- I have a question that isn’t addressed here.
You may ask questions by posting a comment to the KB Robots.txt plugin page. (If you post your support questions as comments below, I probably won’t see them.)
Отличная штука! Помогла мне вернуть Robots.txt, у кого нет роботса из за sitemap. это виртуальный редактор.