WordPress.org

Forums

Sitemap contains urls which are blocked by robots.txt (4 posts)

  1. cjml87
    Member
    Posted 1 year ago #

    Hi there!

    I am very new to wordpress and have had some help installing it. I've downloaded the Yoast SEO plugin, and have been working to submit my sitemap to Google and Bing. However I received these errors this morning and I can't for the life of me figure out what it is, or how to repair this?

    Url blocked by robots.txt.

    Sitemap contains urls which are blocked by robots.txt.

    Sitemap: christinevickers.com/category-sitemap.xml
    Value: http://christinevickers.com/category/engagement-sessions/

    -

    Feb 22, 2014
    Sitemap: christinevickers.com/category-sitemap.xml
    Value: http://christinevickers.com/category/family-lifestyle/

    -

    Feb 22, 2014
    Sitemap: christinevickers.com/category-sitemap.xml
    Value: http://christinevickers.com/category/mini-sessions/

    -

    Feb 22, 2014

    Would really appreciate any help!!
    Thank you!

  2. Mark Senff
    Member
    Posted 1 year ago #

    Your robots.txt file contains this line:

    Disallow: /category/*/*

    This means that search engines are not allowed to search/index anything in the category directory.

    The errors you get basically say "these urls can not be accessed because robots.txt tells me I can't go in that directory" (you'll notice that the urls in question are all in the category directory).

    Easiest way to fix this is to just remove that particular line from robots.txt

  3. cjml87
    Member
    Posted 1 year ago #

    Thank you!

    When I look in my ftp, there is no file robots.txt, where would I find this?

  4. Mark Senff
    Member
    Posted 1 year ago #

    There is probably no physical robots.txt but it's generated dynamically by the Yoast SEO plugin.

    You can change it by going to SEO -> Edit Files.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.