Prevent crawling of internal site search URLs

  • Resolved trmkr

    (@trmkr)


    1 year ago

    Hello,

    I enabled “Prevent crawling of internal site search URLs” setting but there is no change in robots.txt. Can you help please?

    Thank you!

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support devnihil

    (@devnihil)

    1 year ago

    @trmkr If you aren’t seeing your robots.txt as having been updated, it may be due to the file being cached.

    If you clear your cache (server, plugin/theme, browser, or CDN such as Cloudflare) does this resolve the issue? If you are not sure how to clear your server cache you should speak with your host provider.

    We have information on how to clear your browser cache at the following link: https://yoast.com/help/how-to-clear-my-browsers-cache/.

    Thread Starter trmkr

    (@trmkr)

    1 year ago

    Thank you! I discovered that if robots.txt already exists, it is not updated. However, if Yoast creates the file for the first time, these lines are added.

    # START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow: /?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /?rest_route=

Sitemap: https://mywebsite.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

    Regards

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Prevent crawling of internal site search URLs’ is closed to new replies.