Google says Sitemap contains urls which are blocked by robots.txt

Viewing 12 replies - 1 through 12 (of 12 total)
  • Thread Starter zeroonnet

    (@zeroonnet)

    8 years, 4 months ago

    Anyone knows what can it be?

    muchmuch11

    (@muchmuch11)

    8 years, 4 months ago

    Try not to disallow wp-admin.
    Nowdays is useless to disallow wp-admin as by default wordpress already hide unnecessary things from bot.

    Thread Starter zeroonnet

    (@zeroonnet)

    8 years, 4 months ago

    I tried that already.. It’s still the same. Also the robots.txt is being auto generated by wordpress, so it’s wordpress who’s disallowing the wp-admin.

    Anyway it’s strange because I tried without the wp-admin, and also tried “Allow *” and still the same problem.

    Paxjah

    (@paxjah)

    8 years, 3 months ago

    I’m having the same issue, all xml sitemaps created by yoast have URL’s which are blocked by robots.txt

    So I created my own robots.txt file but still the same error, I am now disabling the XML sitemaps from Yoast and creating my own to submit to Google.

    I’ll update if this works, but ideally Yoast should fix this robots.,txt error and also the blank Tools page error.

    This issue is actually quite serious, the robots file currently on Yoast has been preventing Google from indexing anything, I’ve had no click through for over a month and I now find it is due to this, so please do fix it.

    mckanns

    (@mckanns)

    7 years, 6 months ago

    Hi Paxjah,
    Thank you for your comment as I am running into this exact issue. Did your solution work? I have disabled the sitemap and am uploading a ‘clean’ robots.txt file. Question- did you customize your robots.txt file or go with something like this:

    User-agent: *
Disallow:

    Thanks again!
    Shannon

    fortarrigo

    (@fortarrigo)

    7 years, 5 months ago

    I have the same issue.

    My robots.txt allows the crawling of the whole site for every user agent.

    But in Search console all xml sitemaps created by yoast have URLs which are blocked by robots.txt

    I really don’t understand… Is it a Yoast or Google issue?

    Anyone can help me?

    Thanks in advance

    Plugin Support amboutwe

    (@amboutwe)

    7 years, 5 months ago

    @fortarrigo If you have a similar issue, please open a new request. That way it’s easier to track issues and provide assistance. Thanks!

    netstepinc

    (@netstepinc)

    7 years, 5 months ago

    Did this get resolved?
    I’m having the same problem.

    Tradedog

    (@tradedog)

    7 years, 5 months ago

    Me too – same problem, but I’m using All in One SEO. So who knows what the problem is.

    blackbike

    (@blackbike)

    7 years, 5 months ago

    Same here, using WP Plugin, Google XML Sitemaps.
    Any ideas anybody??

    Plugin Support amboutwe

    (@amboutwe)

    7 years, 5 months ago

    While this error is not specific to any plugin, the Yoast knowledge base has an article that provides insight and some suggestions for fixing the problem.

    Fix Blocked Resources In Google Search Console

    blackbike

    (@blackbike)

    7 years, 5 months ago

    Thanks, I think I fixed it by disabling the XML Google Sitemap extension and let Yoast take over the sitemap. Thanks for your reply!

Viewing 12 replies - 1 through 12 (of 12 total)
  • The topic ‘Google says Sitemap contains urls which are blocked by robots.txt’ is closed to new replies.