Is this robots.txt by yoast recommended?

  • Resolved swagatamm

    (@swagatam1975)


    2 years, 5 months ago

    Hello Team,

    Recently when I installed yoast plugin in my website, and checked my robots.txt, I found yoast has changed my default wordpress robots.txt file with the following elements:

    User-agent: *
Disallow: /?s=
Disallow: /page/*/?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /?rest_route=

    I just wanted to confirm if this robots.txt is the best practice and recommended by yoast?

    Thank you for your time.

    Kind Regards

    Swag

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    2 years, 5 months ago

    Hello Swag,

    Thanks for your question about the robots.txt. This isn’t the default directives that would be found in a fresh/default file by Yoast SEO. You can check out this helpful article to see what it should look like. However, if you are convinced that those extra directives are needed, you can leave them as it.

    Thread Starter swagatamm

    (@swagatam1975)

    2 years, 5 months ago

    Thank you so much Maybellyne, for your reply.

    I am not an expert, so I have no idea how to write robots.txt.

    I can assure you that this robots.txt was created by yoast plugin, and this was done not just for one site but for two sites.

    Actually yoast was already installed on my wordpress sites, but to explore other SEO plugins, I temporarily uninstalled Yoast.

    However, I ultimately realized that Yoast is the BEST SEO plugin, therefore I reinstalled Yoast.

    After reinstalling, when I checked my robots.txt file, I found the above mentioned elements in it.

    Previously it looked like this:

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

    After reinstalling Yoast, it changed to this:

    # START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow: /?s=
Disallow: /page/*/?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /?rest_route=

Sitemap: xxxxxxxxxxxxxxxsitemap_index.xml
# ---------------------------
# END YOAST BLOCK

    For further confirmation, you can check my sites with the following URLs:

    homemade-circuits

    .com

    makingcircuits

    .com

    Kind Regards

    Swag

    • This reply was modified 2 years, 5 months ago by swagatamm.
    Plugin Support Maybellyne

    (@maybellyne)

    2 years, 5 months ago

    Since you uninstalled the plugin at some point, other plugins may have added directives to the WordPress dynamic/virtual file at yourwebsite.com/?robots=1. This is what the Yoast SEO plugs into when you are using our plugin.

    But you can edit the file in WordPress > Yoast SEO > Tools > File Editor to match the Yoast SEO guidelines.

    Thread Starter swagatamm

    (@swagatam1975)

    2 years, 5 months ago

    OK, Thank you very much for your kind reply!

    I will try that!

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Is this robots.txt by yoast recommended?’ is closed to new replies.