Posted 1 year ago #
So my google webmaster account is telling me that my robots.txt is disallowing everything... yet it's not. I can either set it to allow (which basically is the same as having no robots.txt) or I can even have NO robots.txt..... and for some reason google thinks my wordpress site has one.... any thoughts on this?
I am allowing things through privacy setting... and my SEO plugin is allowing bots as well... and like I said, I manually removed my robots.txt.
Can we see your robots.txt? Either paste a link (ideally, so we can also check the path) or use a Pastebin.
Posted 1 year ago #
Well right now there is no robots.txt.
It used to say:
User-agent: *
Disallow: /
Then after I was done building I swapped it to:
User-agent: *
Allow: /
Only because I used googles webmaster tools to generate the robots.txt to allow the robots. That's how it does it.
Right now I have completely removed it, as having it say "allow" is just the same.
It says it was read last (via webmaster tools crawler into) 17 hours ago. It updates robots.txt approximately every 24. I'm assuming I have to wait for the next crawl, although I've never run into this issue before.
It used to say:
User-agent: *
Disallow: /Well that explains why you were getting warnings in your Google Webmaster Tools - that code was telling all search engines not to spider your site at all.
You need to wait until Google has re-indexed your site and then see what it says in Webmaster Tools - it should be happier now, though it will be happiest seeing a robots.txt file of some kind. I recommend you put back a default robots.txt - search engines like to see them, even if they don't say much.
This topic has been closed to new replies.
