Yoast SEO
[resolved] [Plugin: WordPress SEO by Yoast] Robots.txt file not updating (6 posts)

  1. Tristan
    Posted 5 years ago #


    I have problem with the robots.txt file when using this plugin.

    As far as I understand, if you make changes to the indexation settings page it should automatically update the robots.txt file right?

    Well, my robots.txt file doesn't change. At first I didn't have one (just the virtual one that WP creates), so I created my own, but none of the disallow rules get added to it when I change it in the plugin.

    Any way to fix this?

    Thanks! :)


  2. Tristan
    Posted 5 years ago #

    I'm also having issues submitting my sitemap to google webmaster... I get this error:

    URL restricted by robots.txt
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

  3. Joost de Valk
    Plugin Author

    Posted 5 years ago #

    This plugin does nothing with your robots.txt file other than allowing you to edit it through the SEO -> Files section. The robots.txt is only one of the ways one can control what a search engine does, and frankly not a very good one.

  4. Tristan
    Posted 5 years ago #

    Hey Joost,

    thanks for the reply :) So when choosing to not have the login pages indexed it just adds a nofollow, noindex to the login page rather than changing the robots file?

    Any idea why the sitemap isn't being accepted? Do you have a link to a good robots.txt file example?



    P.S. - Thanks for making this awesome plugin! It could be easily be sold so really appreciate that you give it to us for free! :)

  5. Tristan
    Posted 5 years ago #

    Works now, logged in today and sitemap was accepted.

    Thanks anyway :)

  6. Joost de Valk
    Plugin Author

    Posted 5 years ago #

    Glad to see you resolved it :)

Topic Closed

This topic has been closed to new replies.

About this Plugin

  • Yoast SEO
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic