WordPress.org

Support

Support » Miscellaneous » Sitemap contains urls which are blocked by robots.txt

Sitemap contains urls which are blocked by robots.txt

Viewing 3 replies - 1 through 3 (of 3 total)
  • Mark Senff
    Participant

    @senff

    Your robots.txt file contains this line:

    Disallow: /category/*/*

    This means that search engines are not allowed to search/index anything in the category directory.

    The errors you get basically say “these urls can not be accessed because robots.txt tells me I can’t go in that directory” (you’ll notice that the urls in question are all in the category directory).

    Easiest way to fix this is to just remove that particular line from robots.txt

    Thank you!

    When I look in my ftp, there is no file robots.txt, where would I find this?

    Mark Senff
    Participant

    @senff

    There is probably no physical robots.txt but it’s generated dynamically by the Yoast SEO plugin.

    You can change it by going to SEO -> Edit Files.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Sitemap contains urls which are blocked by robots.txt’ is closed to new replies.