[resolved] robots.txt Feature works :) but only w WordPress default robots.txt (3 posts)

  1. rrhobbs
    Posted 3 years ago #

    If you (like me) have a custom robots.txt the sitemap exclusions don't work (I'm sure I am not telling you anything you didnt already know)

    If I temporarily rename my custom robots.txt and let WordPress generate robots.txt, I am able to cut / paste the urls to my custom robots.txt.

    I wish there was a way to automate / streamline this?

    I see there is a function that can be used to generate the robots.txt http://codex.wordpress.org/Function_Reference/do_robots however I currently do not have a clue how to utilize it practically i.e. in a (private) page (?)

    That said a feature suggestion for SiteTree might be to somehow have a tab or button so you could see the urls to be added to the robots.txt. But if none of it will work if a custom robots.txt is "live" then maybe it is a moot point.

    Thx again for SiteTree. I am finding it very useful :)


  2. Luigi
    Plugin Author

    Posted 3 years ago #

    that's the reason why I stated "the Robots.txt file created by WordPress" in the tooltip of the functionality you're referring to.

    You don't need to create an actual file to add your own rules to the ones generated by WordPress, nor it's how the default Robots.txt file should be customised (assuming that WordPress has been installed in the root directory).

    do_robots() isn't an end-user function or a template tag, so you cannot and shouldn't use it like that. It actually is the function that generates the Robots.txt file on-the-fly.
    That's where you can "hook" to customise its behaviour or to filter its output but only if you're sure about what you're going to do. Otherwise, go the plugin way.

  3. rrhobbs
    Posted 3 years ago #

    Hi Luigi - thanks for the explanations / clarifications.

    I am neither qualified in most cases nor do I like to mess with core files; it sounds like you are recommending maybe WP Robots Txt Plugin http://wordpress.org/extend/plugins/wp-robots-txt/ to manage the WordPress-generated robots.txt(?).

    Having just tested WP Robots Txt Plugin, I can report it does not currently appear to be working in WP 3.51 :(

    Do you have any suggestions for plugins that might be used alongside SiteTree to manage the robots.txt?


Topic Closed

This topic has been closed to new replies.

About this Plugin

  • SiteTree
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic