WordPress.org

Ready to get started?Download WordPress

Forums

Google XML Sitemaps
[resolved] URL blocked by robots.txt warning msg in Webmaster tools (13 posts)

  1. pmtjohnson
    Member
    Posted 2 years ago #

    Am new to WordPress, and just installing Google XML sitemaps on my new wordpress site, and I'm getting the following error message in Webmaster Tools when doing a sitemap test : URL blocked by robots.txt.
    Would appreciate any help!

    http://wordpress.org/extend/plugins/google-sitemap-generator/

  2. rubixcube6
    Member
    Posted 2 years ago #

    I am having the same problem.

    "Error URL restricted by robots.txt"

    under site configuration->crawler access this is what it says

    User-agent: *
    Disallow: /
    
    Sitemap: http://cdrecords.net46.net/sitemap.xml.gz
  3. arnee
    Member
    Plugin Author

    Posted 2 years ago #

    Yes, you are blocking all search engines from crawling your blog. Check the WordPress privacy settings:

    http://codex.wordpress.org/Settings_Privacy_Screen

  4. rubixcube6
    Member
    Posted 2 years ago #

    my privacy settings are set to "Allow search engines to index this site." still the same problem.

  5. arnee
    Member
    Plugin Author

    Posted 2 years ago #

    Now your robots.txt looks fine, wait 24h before Google downloads it again.

  6. rubixcube6
    Member
    Posted 2 years ago #

    oh ok, Google updated its results. I had to make the robots.txt manually.

  7. pmtjohnson
    Member
    Posted 2 years ago #

    Mine is now fine too! My privacy settings were correct, and the robots.txt file existed, so I've no idea what was happenning, but it seems miraculously to have fixed itself!
    Thanks for the feedback.

  8. entreparentesisupr
    Member
    Posted 2 years ago #

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Sitemap: http://entreparentesis.com/sitemap.xml.gz

    this is what my robots.txt says. It is blocking google to crawl?

  9. bilal4520
    Member
    Posted 1 year ago #

    Same problem here but not resoled.

  10. MelPax
    Member
    Posted 1 year ago #

    No one has an answer to this?

  11. MelPax
    Member
    Posted 1 year ago #

    How is this listed as resolved when there is no answer posted? Some of us are still having issues.

  12. Daniel Dogeanu
    Member
    Posted 1 year ago #

    It seems to be some sort of bugs. I had my website in development and I had the option "Discourage search engines from indexing this site" ticked. When I decided to get my site indexed by Google, surprise, surprise, it told me that robots.txt was blocking the access to the site, even though I knew I unchecked the "Discourage search engines from indexing this site" option and I had the right robots.txt configuration. After that I clicked "Save Changes" several times, I've reset the configuration for XML Sitemaps and completely deleted robots.txt. It immediately worked afterwards.

  13. juerphi
    Member
    Posted 1 year ago #

    I've got the same problem with WP. It seem's that WP doesn't change all settings correctly if I changed the privacy from blocked to public. I solved the problem with a real robots.txt in the document root. This real file has a higher priority. Since I made this everything works fine.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic