WordPress.org

Ready to get started?Download WordPress

Forums

Google XML Sitemaps
[resolved] Robtos.txt blocking all of the pages in sitemap (6 posts)

  1. btzitney
    Member
    Posted 1 year ago #

    Hey guys,

    I just generated a sitemap and submitted it through Google Webmaster Tools. However, there is a warning message for every page included on the site.

    It says the issue is that the URLs are blocked by robots.txt. I don't understand how to fix this.

    My privacy settings are set so google can index the site. My sitemap address is
    http://www.thereisnoquit.com/sitemap.xml
    and my robots.txt address is
    http://www.thereisnoquit.com/robots.txt

    Should I just go in through the FTP and delete the robots.txt file?

    http://wordpress.org/extend/plugins/google-sitemap-generator/

  2. btzitney
    Member
    Posted 1 year ago #

    My main issue is that Google Webmaster tools is saying all the pages on my site are being blocked by robots.txt, but, as you can see in my robots.txt file, none of them are blocked!

    http://www.thereisnoquit.com/robots.txt

  3. entreparentesisupr
    Member
    Posted 1 year ago #

    i Have the same problem. Please can somebody help?!

  4. btzitney
    Member
    Posted 1 year ago #

    I partially fixed the problem. First, I made my own robots.txt file that didn't have any pages besides /wp-admin/ and /wp-includes/ blocked.
    Then I deactivated the plugin and deleted it.
    After that, I reinstalled it and made sure to uncheck the box in the Google XML sitemap configuration page that specifies whether the plugin should make a virtual robots.txt file. After that, I submitted the sitemap through webmaster tools, and 6 of the 8 pages were indexed. Though I have no clue why the other two were not indexed, the home pages and most other pages are now indexed, at least.

    Home that helps.

  5. btzitney
    Member
    Posted 1 year ago #

    I should note that I made sure to uncheck the virtual robots.txt option before I generated the map. Just wanted to make sure that was clear.

  6. Arnee
    Member
    Plugin Author

    Posted 1 year ago #

    The plugin doesn't create the robots.txt file. This file is virtually created by WordPress. The plugin just hooks into this generated robots.txt to include the "Sitemap:..." line. It does not add any Allow/Disallow entries.

    If you don't like the robots.txt generated by WordPress just upload an own (empty) one.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic