Google XML Sitemaps
[resolved] Robtos.txt blocking all of the pages in sitemap (6 posts)

  1. btzitney
    Posted 3 years ago #

    Hey guys,

    I just generated a sitemap and submitted it through Google Webmaster Tools. However, there is a warning message for every page included on the site.

    It says the issue is that the URLs are blocked by robots.txt. I don't understand how to fix this.

    My privacy settings are set so google can index the site. My sitemap address is
    and my robots.txt address is

    Should I just go in through the FTP and delete the robots.txt file?


  2. btzitney
    Posted 3 years ago #

    My main issue is that Google Webmaster tools is saying all the pages on my site are being blocked by robots.txt, but, as you can see in my robots.txt file, none of them are blocked!


  3. entreparentesisupr
    Posted 3 years ago #

    i Have the same problem. Please can somebody help?!

  4. btzitney
    Posted 3 years ago #

    I partially fixed the problem. First, I made my own robots.txt file that didn't have any pages besides /wp-admin/ and /wp-includes/ blocked.
    Then I deactivated the plugin and deleted it.
    After that, I reinstalled it and made sure to uncheck the box in the Google XML sitemap configuration page that specifies whether the plugin should make a virtual robots.txt file. After that, I submitted the sitemap through webmaster tools, and 6 of the 8 pages were indexed. Though I have no clue why the other two were not indexed, the home pages and most other pages are now indexed, at least.

    Home that helps.

  5. btzitney
    Posted 3 years ago #

    I should note that I made sure to uncheck the virtual robots.txt option before I generated the map. Just wanted to make sure that was clear.

  6. Arne Brachhold
    Plugin Contributor

    Posted 3 years ago #

    The plugin doesn't create the robots.txt file. This file is virtually created by WordPress. The plugin just hooks into this generated robots.txt to include the "Sitemap:..." line. It does not add any Allow/Disallow entries.

    If you don't like the robots.txt generated by WordPress just upload an own (empty) one.

Topic Closed

This topic has been closed to new replies.

About this Plugin

  • Google XML Sitemaps
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic