Update: I’ve noticed that when we clear the cache (WP Rocket), we lose all of our robots.txt customizations.
Second issue:
We migrated from Yoast, and never had an issue with Google fetching the robots.txt file. However, when using Google Webmaster testing tool (https://www.google.com/webmasters/tools/robots-testing-tool), to update the file in Google, we receive the following error:
robots.txt fetch failed
You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.
The file is accessible at its origin point (domain.com/robots.txt), and we’ve entered the URL into the Google Robots Testing Tool to see if GoobleBot is blocked, and it is Allowed.
Can you shed light on this issue and a fix?
@bella2011 I do know of some hosting providers that generate their own robots.txt file, even if you delete it in the site root directory. May I ask which hosting provider you’re using for this website?
Also, can you share the URL to your robots.txt. Unfortunately we’re not able to provide much help without access to the page.
Hi there,
So, we talked with our hosting provider – they do not generate the robots.txt file, they must be manually added in the root directory.
I’ve recreated the issue again, after confirming the removal of the robots.txt file from the root.
When changes are made inside the AIOSEO dashboard to robots.txt file and saved. I then hit Clear Cache and all of the updates are erased.
I’ve read your post on Cloudflare Firewall rules (https://aioseo.com/docs/aioseo-uses-rest-api/) and updated Cloudflare.
So the conflict is with our Caching plugin (WP-Rocket).
Have you had prior issues with caching plugins and is there a fix?
Thanks.
Update:
I have fixed the Error:robots.txt fetch failed in Google Webmaster testing tool.
The only error that remains with the WP Rocket Caching plugin, which erases any changes made to the Robots.txt file in the AIOSEO dashboard, after updates have been saved.
@bella2011 if you disable WP Rocket and save, the changes do not go away, correct?
