Hello @monicatreanor
Thanks for reaching out about your robots.txt file. We have addressed your concerns below:
Failed: Robots.txt unreachable
Before Google crawled the pages of your site, they tried to check your robots.txt file to ensure they didn’t crawl any pages that you had roboted out. However, your server returned a 5xx (unreachable) error when they tried to retrieve your robots.txt file.
To make sure they didn’t crawl any pages listed in that file, they postponed the crawl. Your hosting provider may be blocking Googlebot, or there may be a problem with the configuration of their firewall.
Note that if the contents of the robots.txt file are different in your browser than what Google sees, work with your hosting company to remove any server rules that might show different robots.txt content to different user agents.
No File Editor option
The Yoast SEO plugin includes the ability to create and edit the robots.txt file for your site when your site meets the following requirements:
- Enable file editing, which is the default option for most WordPress installs
- Set server-level permissions to allow writing to files, specifically the robots.txt file.
Since you do not see the option to create or edit the file, WordPress has disabled file editing, or your server doesn’t allow writing to the file. Your web host can help you enable these features if you want to use Yoast SEO to edit the robots.txt file. Alternatively, you can always create or edit the file at the server level.
