Support » Plugin: Yoast SEO » Google Search console blocked through robots.txt

  • Resolved schwabentraum

    (@schwabentraum)


    Hello,

    yesterday I launched our new blog and I have issue with indexing the blog. I installed the YOAST SEO followed the wizard and set the first question to “Index the page”. After that I created a property at the google search console, verified that via the new DNS-method and tried to add the sitemap. The follow error message appeared after I tried to investigate what the problem is. This message is from the live url check of the search console.

    Crawling allowed? No: Blocked by robots.txt
    Access: Error: Blocked by robots.txt

    I allowed everything in the robots.txt.. So I absolutely don’t know why this happens. In the Google Forum the guys told me to write here, because this is a known problem with WordPress and Yoast, but I didn’t find a solution..

    The robots.txt: https://schwabentraum.com/robots.txt
    The Sitemap: https://schwabentraum.com/sitemap_index.xml

    Thank you alot! I absolute don’t know what to do else..

    Best regards,
    Daniel from Schwabentraum

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support Pcosta88

    (@pcosta88)

    Hi Daniel,

    1. Robots Files
    We suggest having the default rules, as if Google needs to access resources in those files certain disallow rules prevent Google from doing so. This means Google does not have all the information to understand and then index the page. This guide explains more: https://yoast.com/wordpress-robots-txt-example/ and this guide explains how to edit the robots.txt: https://kb.yoast.com/kb/how-to-edit-robots-txt-through-yoast-seo/.

    You can use the defaults below.

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Sitemap: https://schwabentraum.com/sitemap_index.xml

    2. Yoast Sitemap
    The sitemap part of the robots rules above, assumes you have enabled the Yoast Sitemap. If you have not, please do so. This guide explains more: https://kb.yoast.com/kb/enable-xml-sitemaps-in-the-wordpress-seo-plugin/

    Plugin Support amboutwe

    (@amboutwe)

    This thread has been marked as resolved due to lack of activity.

    You’re always welcome to re-open this topic. Please read this post before opening a new request.

    Thanks for understanding!

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.