Hi !
I read this today too and they said that we need to remove Disallow: / wp-includes / from robots.txt
I’m waiting confirmation from Yoast SEO developpers
Hi,
Had similar emails re half of my sites. All that is in my robots.txt is
User-agent: *
Disallow: /wp-admin/
When I used “fetch as googlebot” for smartphone and rendered there were many stylesheets that could not be read and js script.
Weirdly, when I went to file-editor in Yoast SEO tools it said I didn’t have a robots.txt file!!!
Any advice?
The same here but I have a robots.txt on my yoast SEO I don’t know how to test if everything is fine on my pages !! When I test robots.txt I have green message allow !!
Also having the same issue. I’ve not touched anything.
We await a response from the developers Yoast SEO.
Hi,
Had similar emails re half of my sites. All that is in my robots.txt is
User-agent: *
Disallow: /wp-admin/
When I used “fetch as googlebot” for smartphone and rendered there were many stylesheets that could not be read and js script.
Weirdly, when I went to file-editor in Yoast SEO tools it said I didn’t have a robots.txt file!!!
Any advice?
EXACTLY the same for me: not all my sites, and I only have a disallow on the /wp-admin/
According to Yoast, even that disallow on wp-admin is a waste of space, read here. Strange that the SEO plugin from Yoast does make this file though, as far as I can tell.
And even weirder: where to edit the file? Because when I ‘edit’ robots.txt in Google Search Console then the tool says its fine, but where to edit this robots.txt file that, according to the wisdom of the internet, is a virtual file that does not really exist?
Even stranger: in Search Console you can fetch and display your page like Google sees it and how a human visitor sees it. According to this tool, the Adsense banner I have on my homepage cannot be ‘seen’ by Google crawlers yet it can be seen by humans, now, Adsense == Google, so why is it blocking it’s own crawlers?
*completely confused*
so nobody from Yoast reponse here !!!
Yesterday I removed Disallow: /wp-includes/ from robots.txt like yoast suggested ! but I still see it today nothing changed on my file !! is the plugin SEO yoast work fine ??
@fritsje said: Strange that the SEO plugin from Yoast does make this file though, as far as I can tell.
The plugin does not (unless you explicitly do it – see below). WordPress itself makes it. Deactivate Yoast SEO and then check the “Discourage search engines from indexing this site” box at: Settings > Reading > Search Engine Visibility
WordPress makes an on-the-fly robots.txt viewable at your-domain/robots.txt and the content will be:
User-agent: *
Disallow: /
This blocks the entire site. If you uncheck the box, WordPress changes the robots.txt (at least on my WP 4.2.3 system) to:
User-agent: *
Disallow: /wp-admin/
However, no actual robots.txt file will be written to the server root. Go look. No file. It is created on-the-fly and you can find elsewhere on the forums where the core code is shown that does it.
Yoast advises against blocking /wp-admin/ in a 5 February 2015 post: https://yoast.com/wordpress-robots-txt-example/ . So WordPress is apparently doing the wrong thing now. My solution is to make a real robots.txt file at the site root, which will override the on-the-fly robots.txt. Use the following content to allow access to your entire site:
User-agent: *
Allow: /
You can do this with your favorite text editor and an SFTP tool, or you can use the “SEO > Tools > File editor” included in Yoast SEO 2.3.2. This reads what WordPress is creating on-the-fly, and it will let you change it and save a custom robots.txt file. When you hit the “Save changes to robots.txt” button, a real robots.txt file will be written to your site root (which overrides the on-the-fly version).
So the question remains, why is WordPress still blocking access to /wp-admin/?
Thanks thenightrider, I got it now.
I had Yoast SEO create Robots.txt for me. It came up with:
User-agent: *
Disallow: /wp-admin/
I changed it to:
User-agent: *
Allow: /
