Title: Robots.txt `host` directive
Last modified: August 21, 2016

---

# Robots.txt `host` directive

 *  [dnk6742](https://wordpress.org/support/users/dnk6742/)
 * (@dnk6742)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/)
 * First, thanks for a great CMS.
 * Second, i suggest to add an option to add custom lines to robots.txt file somewhere
   on a settings page.
 * In addition, there is a good idea to put `Host` directive in robots.txt^ it is
   required for some search engines (like Yandex.ru) and can result in site being
   banned if there are too many cnames/aliases pointing to one real site.
 * I found the temporary solution in the following:
    functions.php:1108:
 *     ```
       $output .= "\n";
       $output .= "Host:". parse_url ( get_option ('siteurl'), PHP_URL_HOST )."\n";
       ```
   

Viewing 7 replies - 1 through 7 (of 7 total)

 *  [Clayton James](https://wordpress.org/support/users/claytonjames/)
 * (@claytonjames)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566404)
 * >  i suggest to add an option to add custom lines to robots.txt file somewhere
   > on a settings page.
 * I think if you create an actual robots.txt file and place it on your web space,
   the virtual version will be ignored.
 * >  I found the temporary solution in the following:
   >  functions.php:1108:
 * You should never edit WordPress core files. Changes to core files will be lost
   when you update.
 * Some reference material: [Search Engine Optimization for WordPress](http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress)
 *  Thread Starter [dnk6742](https://wordpress.org/support/users/dnk6742/)
 * (@dnk6742)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566435)
 * ClaytonJames, thank you for reminding me of this 🙂
    Having a hand-written robots.
   txt is a solution, yet that is ideologically wrong, as I can see. This option
   is worth having, I think.
 *  [Clayton James](https://wordpress.org/support/users/claytonjames/)
 * (@claytonjames)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566443)
 * > Having a hand-written robots.txt is a solution, yet that is ideologically wrong,
 * I have no desire to change your opinion, but I would be interested in knowing
   why you feel a hand-written robots.txt file is wrong.
 *  Thread Starter [dnk6742](https://wordpress.org/support/users/dnk6742/)
 * (@dnk6742)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566482)
 * The first example crossing my mind is the situation when you can’t access FTP
   for whatever reason. In this case – and if robots.txt is written by WP – having
   an access to WP admin panel is enough.
    Then, keep in mind that the directive“
   Ask search engines not to index the site” will not work if robots.txt exists.
 *  [Clayton James](https://wordpress.org/support/users/claytonjames/)
 * (@claytonjames)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566493)
 * Interesting perspective.
 *  Moderator [Ipstenu (Mika Epstein)](https://wordpress.org/support/users/ipstenu/)
 * (@ipstenu)
 * 🏳️‍🌈 Advisor and Activist
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566540)
 * > Second, i suggest to add an option to add custom lines to robots.txt file somewhere
   > on a settings page.
 * How about I meet you guys halfway?
 * [http://wordpress.org/plugins/wp-robots-txt/](http://wordpress.org/plugins/wp-robots-txt/)
 * (Also [http://wordpress.org/plugins/wordpress-seo/](http://wordpress.org/plugins/wordpress-seo/)
   does that)
 *  Thread Starter [dnk6742](https://wordpress.org/support/users/dnk6742/)
 * (@dnk6742)
 * [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566541)
 * First of all, thanks. Second, I believe this should be endine functionality, 
   at least to certain extent.
 * But yes, thanks for the plugin again 🙂

Viewing 7 replies - 1 through 7 (of 7 total)

The topic ‘Robots.txt `host` directive’ is closed to new replies.

## Tags

 * [robots.txt](https://wordpress.org/support/topic-tag/robots-txt/)

 * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/)
 * 7 replies
 * 3 participants
 * Last reply from: [dnk6742](https://wordpress.org/support/users/dnk6742/)
 * Last activity: [12 years, 3 months ago](https://wordpress.org/support/topic/robotstxt-host-directive/#post-4566541)
 * Status: not resolved

## Topics

### Topics with no replies

### Non-support topics

### Resolved topics

### Unresolved topics

### All topics