Big issue with robots.txt

Resolved marvindepagamon
(@marvindepagamon)

7 years, 2 months ago

I have a SEO issue on my wordpress website : http://www.pagamon.com

Depending on which browser I am using I see that the robots.txt is either :

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

or

User-agent: *
Disallow: /

But in fact it appears that Google bots do not have access to the website at all.
I created a robots.txt file with Yoast. I can see it with my FTP and it is :

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Allow: /wp-content/uploads/

Why is this file not overriding WordPress automatic robots.txt ?

Thanks in advance for your help. I’m a bit desperate after trying so many things to fix it.

Marvin

The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)

Jeroen Rotty
(@jeroenrotty)

7 years, 2 months ago

Hi,

If Yoast SEO can’t update your robots.txt file, it’s due to lacking file permissions. You will be better off editing it directly through (S)FTP or control panel with your hosting provider. When needed, please reach out to your hosting company for assistance.

Jeroen

Thread Starter marvindepagamon
(@marvindepagamon)

7 years, 2 months ago

I did try to edit it through FTP. The file currently in the directory that I can see through FTP is correct but in some way WordPress has the lead and WordPress commands seem to override my robots.txt file
Saša
(@stodorovic)

7 years, 2 months ago
It seems that you have custom PHP script for robots.txt:
```
$ curl -i http://www.xxxxxx.com/?robots=1
HTTP/1.1 200 OK
Date: Tue, 02 Apr 2019 08:26:36 GMT
Server: Apache
X-Powered-By: PHP/7.3.0
X-Robots-Tag: noindex, follow
...
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

$ curl -i http://www.xxxxxx.com/robots.txt
HTTP/1.1 200 OK
Date: Sun, 17 Mar 2019 21:45:11 GMT
Server: Apache
X-Powered-By: PHP/5.4.45
X-Robots-Tag: noindex, follow
...

User-agent: *
Disallow: /
```
Please notice differences in X-Powered-By. You should ask your host for more details. You need to find “old PHP code” which blocks access.
Thread Starter marvindepagamon
(@marvindepagamon)

7 years, 2 months ago

Thanks a lot, I guess with this I will be able to fix it

devnihil
(@devnihil)

7 years, 2 months ago

@marvindepagamon Thanks for confirming that you can resolve this by addressing the custom script. We are going ahead and marking this issue as resolved. If you require any further assistance please create a new issue.