Just for clarification, the entire URL for my sitemap is. thetexasreport(dot)com/sitemap.xml
Also I used the Arne Brachhold Google sitemap generator.
anyone have any thoughts on this by chance?
The only thing they could derive from this issue is that my .htaccess is causing the robots.txt to write automatically.
Huh???
I attempted to submit my sitemap to Google – however they came back and said it is being blocked by my robots.txt. There is no robots.txt on my server.
Are you sure that it is not actually a “robots.txt unreachable” error?
http://www.thetexasreport.com/robots.txt
there’s either one there, and you missed it, or youve got a setting or a plugin, during the work.
gee, I refreshed it, and the text changed. who woulda thunk.
This is the error – I am getting from google.
URL restricted by robots.txt
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit
Also, why would the path to my robots.txt be valid when there is NONE showing on my server.
http://www.thetexasreport.com/robots.txt
The support repts at GoDaddy (after 45 minutes of studying it) said it was a problem with the .htacess that is automatically generating the .robots.txt.
This is insane.
Well, I tried makng one minor change to the robots.txt. . . that’s why the text changed.
I put it back to default so I don’t confuse anyone.
Let me reprase that. . . I uploaded a robots.txt to triy and see if I could overwrite one not showing on the server.
This didn’t help.
is there something I am missing or are you just not getting this:
you said this, in your first post:
Remember, I do not have a visible robots.txt file anywhere on my server. I’ve went through every folder.
Ive just showed you, that you do. And I indicated watching the text change on a reload of the page.
Then, you say this:
Well, I tried makng one minor change to the robots.txt. . . that’s why the text changed.
I put it back to default so I don’t confuse anyone.
Do you not see the contradictory statements here?
What are you trying to do?
If you want google to not be blocked, remove that file. OR — fix it so googlbot has access. You made that change once, you said it above, and I saw it. But then you changed it back so not to confuse anyone??
If you uploaded a robots.txt ..
Check that the version of wordpress you are using does not have the privacy option ticked on. Next is to look at your plugins, particularly SEO plugins.
Like I said earlier, theres only 3 potential causes of this: a robots.txt existing on your server, the wordpress privacy option, or a plugin setting.
# 1 – I know the robots.txt file can be viewed. I saw the text too.
However, I cannot find the file ANYWHERE on my server.
# 2 – After I made the change to the NEW robots.txt file I uploaded it just to see if I could overwrite one I was missing – this did not fix the problem. I still got the same error from Google after I re-submitted the sitemap.
Not trying to make this confusing.
What am I trying to do? I want to remove the robots.txt file from my server so I can submit my sitemap.
However, GoDaddy even looked for it for 45 minutes and they could not find it. Even though we could view the text.
This is why it’s so strange.
its not strange.
WP has a privacy setting. Ive mentioned it 3x times now.
Have you checked that?
Have you checked all of your plugin settings?
Yes, I have checked the privacy settings.
I have checked the plugin settings to the best of my ability. . . not really sure what I am looking for on these.
I think it’s strange because my host support looked at it for 45 minutes and they were baffled. And, I’ve mentioned it multiple times as well.
I’m not an IT guy, so I’m really trying to dig in and see what’s going on here.
This is still extremely confusing whats going on here.
Anyone want to give this a shot?
Look, this is not that complicated.
The robots.txt is being generated by WordPress. It is not a file on the server, WordPress is creating it on the fly, based on your settings on the Settings->Privacy page.
Because you have set it to be not private, the robots.txt you currently have contains this:
User-agent: *
Disallow:
This robots.txt basically says “Allow everything”. Google will not block because of this. However, Google may have your old robots.txt cached, and will need to refresh before you can do anything with it. If at any given point in the past you had it set to “Private”, then Google may have noticed it then and will need time before it gets around to rechecking your site. Until that happens, you can’t do anything.
In other words, you don’t need to *do* anything. Your site is now correct. Google needs to notice that, and you need to wait for it to do so.
Thank You Otto42 for giving me a straight answer.
I understand.
