Sitemap contains urls which are blocked by robots.txt.

  • Resolved graphicsxp

    (@graphicsxp)


    7 years, 1 month ago

    I get this error in Google Search Console:

    Sitemap contains urls which are blocked by robots.txt.

    This happened since I’ve migrating my website from http to https. It makes no sense because robots.txt allows all urls to be indexed – see for yourself https://www.pretty-story.com/robots.txt

    Also I’ve noticed that when I request the sitemap.xml in the URL, the content-type in the response is text/html. Therefore it is not showing as XML in the browser.

    What could be the root of all these issues ?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support amboutwe

    (@amboutwe)

    7 years, 1 month ago

    The second to last line is Disallow: /. This blocks access to all bots, including Google.

    Thread Starter graphicsxp

    (@graphicsxp)

    7 years, 1 month ago

    sorry, I posted the link to my sandbox website…. 🙁

    the actual website is [https://www . joliehistoire . com]

    After more investigation, I can tell from my Apache log that Googlebot is crawling my website and gets a 200.
    However Search Console is still showing these error messages. I don’t get it.

    Plugin Support amboutwe

    (@amboutwe)

    7 years, 1 month ago

    I don’t see an immediate reason as to why Google can’t reach your site either. Sometimes Google Search Console takes up to a week to recrawl your site. Sometimes you can speed up this process as described in this KB: Fetch as Googlebot

    If the error remains after a week, please contact Google for advice here: https://support.google.com/

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Sitemap contains urls which are blocked by robots.txt.’ is closed to new replies.