Support » Plugin: Yoast SEO » Robots.txt is blocking crawling, but I don’t have a robots.txt

  • Resolved thegenxjourney

    (@thegenxjourney)


    I installed the Yoast SEO plugin yesterday and today got an email from google saying that my robots.txt was blocking crawling.

    I never setup a robots.txt for this website and logged in via Filezilla and didn’t see one in there.

    I logged in to WordPress and it is not blocked there. The only thing that changed was installing Yoast so I looked around the FAQ and it said to go into tools and edit the robots.txt. It tells me there isn’t one.

    If Yoast says that I don’t have a robots file, but google says I do, I am not sure how to proceed.

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support devnihil

    (@devnihil)

    We checked on your site and do see that a robots.txt file is being served from http://www.thegenxjourney.com/robots.txt:

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    However, the directives in your robots.txt file are fairly standard so should not be causing Google to be blocked from crawling your site. We also checked your robots.txt file in a test utility here, and also checked the pages headers for any X-Robots tags set to ‘noindex’ and none were detected.

    If you are receiving a warning that Google is unable to crawl a URL on your site due to the ‘URL blocked / restricted by robots.txt’this can be caused by several possible causes and we have a guide on resolving them here:
    https://kb.yoast.com/kb/crawl-error-url-restricted-by-robots-txt/

    Plugin Support amboutwe

    (@amboutwe)

    This thread has been marked as resolved due to lack of activity by original poster.

    If you are not the original poster but have a similar issue, please open a new request.

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.