WordPress.org

Forums

WordPress SEO by Yoast
[resolved] Sitemap blocked by robots.txt - already checked my robots file. (9 posts)

  1. techluminati
    Member
    Posted 3 years ago #

    I got this error message from google when I tried to submit the index_sitemap.xml file in google webmaster tools:
    "Sitemap contains urls which are blocked by robots.txt."

    my robots file is at http://techluminati.com/robots.txt

    previously, I did not have a robots.txt file, yet wordpress seemed to have one which I could not find anywhere on my website.
    so I created a new robots.txt file and uploaded it to the server.

    I have also checked the option in wordpress seo to generate sitemaps automatically, and I believe its doing a great job..although for some reason the sitemaps are not being accepted by google.

    Can anybody please help?

    WordPress version: 3.3.2
    WordPress SEO version: 1.1.9

    http://wordpress.org/extend/plugins/wordpress-seo/

  2. Joost de Valk
    Member
    Plugin Author

    Posted 3 years ago #

    Delete it, the default WordPress robots.txt file is better...

  3. techluminati
    Member
    Posted 3 years ago #

    Hello Joost,

    Thank you very much for taking the time to reply.
    I have deleted it and now its back to the default.

    However, Google still cannot get the sitemap, it says the access is being blocked by robots.txt

  4. techluminati
    Member
    Posted 3 years ago #

    Any updates on what should I do next? as mentioned earlier, google is still begin blocked by robots.txt
    thanks...

  5. Joost de Valk
    Member
    Plugin Author

    Posted 3 years ago #

    Can you copy paste the message from Google? I'm quite sure it says SOME URL's are blocked.

  6. techluminati
    Member
    Posted 3 years ago #

    Hello Joost,

    Thank you for your reply,
    here is the exact error message from google webaster tool's sitemap test section.
    "Sitemap contains urls which are blocked by robots.txt."

    and the following column says - issues count - 2
    following which:
    Example - http://techluminati.com/category-sitemap.xml
    http://techluminati.com/post-sitemap.xml

  7. laurasbadideas
    Member
    Posted 3 years ago #

    Google doesn't reread the robots.txt file when you submit a sitemap; it uses a cached value of robots.txt from the last time it crawled your site. So the message may just mean that some of the URLs in your sitemap were blocked by your old robots.txt. You can find the date that Google last read your robots.txt by going to Health / Blocked URLs on Webmaster Tools.

  8. techluminati
    Member
    Posted 3 years ago #

    Hello there,

    That was a perfect answer! I let it wait a while to get the new robots.txt, and now the sitemap tests are successful, I am no longer getting the error messages.

    Thanks a lot to Laura and Joost (great fan of your work btw)

  9. Sapphire
    Member
    Posted 2 years ago #

    I'm getting the same warning, and waiting for the robots.txt file to get updated didn't help. Specifically, it's telling me it the sitemap contains links to my images, and it can't reach them. But (a) the SEO plugin is set to noindex my images, (b) when I actually search the sitemap itself, the example URLs WMT lists as blocked aren't even in there and (c) the wp-content folder is not blocked (nor are any of the folders below it). I have no idea what's going on here!

    Also, WMT reports that the sitemap is giving it about 3 times as many images as I actually have in the Media Library. It seems to be accessing some really old images that I deleted last year. Maybe they're still lurking on the server somewhere, but I don't understand how it's accessing them at all.

Topic Closed

This topic has been closed to new replies.

About this Plugin

  • WordPress SEO by Yoast
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic

Tags

No tags yet.