WordPress.org

1. do77
   Member
   Posted 3 years ago #

   Hi everyone,

   I successfully created the sitemap for my page with this plugin but I am having some troubles with my robots.txt file. In my plugin panel it says
   

   The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!

   So I didn't create another robots.txt in my root. But when I go to my webmaster-tools account google says that Googlebot was blocked by my url and couldn't crawl my content. My virtual robots.txt blocks google completely.

   Where can I change this virtual robots.txt or how can I disable the file thus I can upload a regular one. Hope someone has an advice,

   Thanks!
   dk

2. matinciel
   Member
   Posted 3 years ago #

   Hi,
   I do have the same problem. Please let me know if you find a solution.
   I begin my search yet :-p .

3. behanj
   Member
   Posted 3 years ago #

   me too, getting 401/407 authentication error in google webmaster tools

4. behanj
   Member
   Posted 3 years ago #

   Just found this, not WordPress specific but may be related
   http://forums.digitalpoint.com/showthread.php?p=8719602

5. behanj
   Member
   Posted 3 years ago #

   This is still a problem for me.
   After 4 days Google still hasn't indexed my site, all 5 pages.
   The 401/407 error persists.
   Can I disable the virtual robots.txt file as it seems to be messing the whole thing up?

6. do77
   Member
   Posted 3 years ago #

   Hey guys,

   a little late but someone might still hasn't figured it out. Create a file named robots.txt and upload it to your root. Google automatically prefers this one and your problem should be fixed. At least its working for me

7. do77
   Member
   Posted 3 years ago #

   Hey guys,

   a little late but someone might still hasn't figured it out. Create a file named robots.txt and upload it to your root. Google automatically prefers this one and your problem should be fixed. At least its working for me

8. Arne
   Member
   Posted 2 years ago #

   If the contents of your WordPress-generated virtual robots file block Google, check your WordPress privacy settings. There is an option in the admin panel to tell search engines NOT to crawl your site. Probably you have this option checked :-)

9. do77
   Member
   Posted 2 years ago #

   Thanks arnee!
   What the heck, Ive never check the privacy settings and I seriously blocked all search engines :-D

10. philipanderson
    Member
    Posted 2 years ago #

    I'm still confused . . . if this plugin is enabled, must I completely delete the robots.txt from the root of my domain if wordpress is installed in the root of my domain. What is the effect of having a robots.txt in the blog root and having the XML Sitemap plugin installed.

    Tks,

    Phil.

11. tanchiahuei
    Member
    Posted 2 years ago #

    THANKS ARNEE!
    I spent all day trying to figure out why my new live site was now blocking google. Turns out I *did* switch on the block search engine feature in wordpress during the preparation of the new site (since it was a duplicate of the site that was online).
    Thanks for the reminder :)

12. parisvega
    Member
    Posted 2 years ago #

    I've unchecked the privacy setting. I'm still getting blocked. I'm using WebFaction for hosting. Any ideas?

13. mcbrowne
    Member
    Posted 2 years ago #

    Tried the above.

    The privacy setting on my site is allow all and my robots.txt says:

    User-agent: *
    Allow: /

    But google fetch says access denied by robots.txt

14. Arne
    Member
    Posted 2 years ago #

    @philipanderson: No problem, you can add the following to your custom robots.txt file:

    Sitemap: http://url/to/sitemap.xml.gz

    @mcbrowne: Can you check if there is a static robots.txt file in your blog root?

15. twmeggs
    Member
    Posted 2 years ago #

    Hi,
    Have used this plugin successfully before on another blog. Have just installed on a new blog today here and am having some issues.

    I have built the sitemap successfully and I see the two sitemap files in the root of my website directory. However, when I submit the sitemap url to the Google Webmaster tool, I get the following error after submitting and waiting 5-10mins:

    "URL restricted by robots.txt

    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit."

    I do not have a robot.txt file in the root of this site. However, when I now look at http://www.passiveincometrial.com/robots.txt it produces:

    "User-agent: *
    Disallow:
    Sitemap: http://passiveincometrial.com/sitemap.xml.gz
    "
    which, I assume, is what is blocking the Googlebot from looking at the sitemap.

    I'm sure this is user error on my part somehow, but if someone could point out my mistakes in setting this up, I would greatly appreciate it.
    Thanks
    Tim

16. twmeggs
    Member
    Posted 2 years ago #

    and interestingly when I look at:

    http://www.passiveincometrial.com/robots.txt

    I see:

    User-agent: *
    Allow: /

    but Google is still saying the robots.txt file is restricting their access.

    Any help would be greatly appreciated. Thanks

17. twmeggs
    Member
    Posted 2 years ago #

    Possibly resolved this. Think it might be down to my WordPress settings having previously been set to PRIVATE on the last time Google crawled the site. Now that setting has been updated, I think that once they crawl again and see the new robots.txt file, the problem should disappear.

    Will post again if this is the case.
    Thanks

18. Arne
    Member
    Posted 2 years ago #

    Since this was the problem many times, the sitemaps plugin will warn you in the administration page if the privacy setting of WP is enabled since version 3.1.8 :-)

19. parisvega
    Member
    Posted 2 years ago #

    To solve my problem I simply created a file named robot.txt and placed it in my root directory with the following text inside of it:

    User-agent: *
    Allow: /

    That's it. Problem solved.

20. pbgswd
    Member
    Posted 2 years ago #

    Hi,
    this is a problem that is very trivial but also very stupid. Why why why is my robots.txt file getting overwritten? Does WordPress core think it should do this because I dont want anybody to visit my website?

    I have had to go into the file and update it more than once.

    Upon the robots.txt file, I have changed permissions:
    chmod 744 robots.txt
    (Only I can edit, everyone else can read only).

    I have to keep watching this now to make sure it isnt still happening.

Topic Closed

This topic has been closed to new replies.