Support » Plugin: SiteTree » robots.txt Feature works :) but only w WordPress default robots.txt

  • Resolved rrhobbs


    If you (like me) have a custom robots.txt the sitemap exclusions don’t work (I’m sure I am not telling you anything you didnt already know)

    If I temporarily rename my custom robots.txt and let WordPress generate robots.txt, I am able to cut / paste the urls to my custom robots.txt.

    I wish there was a way to automate / streamline this?

    I see there is a function that can be used to generate the robots.txt however I currently do not have a clue how to utilize it practically i.e. in a (private) page (?)

    That said a feature suggestion for SiteTree might be to somehow have a tab or button so you could see the urls to be added to the robots.txt. But if none of it will work if a custom robots.txt is “live” then maybe it is a moot point.

    Thx again for SiteTree. I am finding it very useful 🙂

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author Luigi


    that’s the reason why I stated “the Robots.txt file created by WordPress” in the tooltip of the functionality you’re referring to.

    You don’t need to create an actual file to add your own rules to the ones generated by WordPress, nor it’s how the default Robots.txt file should be customised (assuming that WordPress has been installed in the root directory).

    do_robots() isn’t an end-user function or a template tag, so you cannot and shouldn’t use it like that. It actually is the function that generates the Robots.txt file on-the-fly.
    That’s where you can “hook” to customise its behaviour or to filter its output but only if you’re sure about what you’re going to do. Otherwise, go the plugin way.

    Hi Luigi – thanks for the explanations / clarifications.

    I am neither qualified in most cases nor do I like to mess with core files; it sounds like you are recommending maybe WP Robots Txt Plugin to manage the WordPress-generated robots.txt(?).

    Having just tested WP Robots Txt Plugin, I can report it does not currently appear to be working in WP 3.51 🙁

    Do you have any suggestions for plugins that might be used alongside SiteTree to manage the robots.txt?


Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt Feature works :) but only w WordPress default robots.txt’ is closed to new replies.