Support » Plugin: Yoast SEO » Asking Yoast to look into my Robots.txt?

  • Resolved marbs

    (@marbs)


    I want to ask the experts (Yoast) if my Robots.txt is not preventing my site to rank well due to my robots.txt file?

    My Robots.txt file is:

    User-Agent: *
    Allow: /wp-content/uploads/
    Allow: /ads.txt
    Allow: /wp-admin/admin-ajax.php
    Disallow: /wp-content/plugins/
    Disallow: /wp-admin/
    Disallow: /readme.html
    
    Sitemap: https://mydomain.com/sitemap_index.xml
    
    User-agent: Twitterbot
    Disallow: *
    
    Allow: /wp-content/uploads/*

    Waiting for your reply,

    Thanks

Viewing 7 replies - 1 through 7 (of 7 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    Hello Marbs,

    Thanks for reaching out regarding your robots.txt file.

    Indeed, any mistakes you make in your robots.txt can seriously harm your site. For most WordPress sites, the following example is best practice:

    # This space intentionally left blank
    # If you want to learn about why our robots.txt looks like this, read this post: https://yoa.st/robots-txt
    User-agent: *

    We even use this approach in our own robots.txt file. These help articles will be beneficial:

    https://yoast.com/ultimate-guide-robots-txt/
    https://yoast.com/wordpress-robots-txt-example/

    Thread Starter marbs

    (@marbs)

    Hi @maybellyne thanks for helping me out. So I have changed my robots.txt into this. Is that ok?

    # This space intentionally left blank
    # This space intentionally left blank
    User-agent: *
    
    # blocking search results
    Disallow: /?s=*
    Disallow: /search/*
    • This reply was modified 3 months, 4 weeks ago by marbs.
    Plugin Support Maybellyne

    (@maybellyne)

    Thanks for following up.

    Are you trying to block your site’s search result pages from being indexed? Since you are using our plugin, you’re fine. We’ve got you covered. Your internal search results pages are noindexed by default in Yoast SEO. I do not think it is necessary to have extra directives that cover this in your robots.txt. You can learn more at https://yoast.com/blocking-your-sites-search-results/

    Thread Starter marbs

    (@marbs)

    @maybellyne I actually want to be found. I am trying to recover on search engine Google after using Rank Math.

    Thread Starter marbs

    (@marbs)

    So I don’t have to include the search. My robots.txt should be look like this to be discovered by search engines?

    # This space intentionally left blank
    # This space intentionally left blank
    User-agent: *
    Plugin Support Jeroen Rotty

    (@jeroenrotty)

    Hey @marbs,

    Ah okay. Yes, looks good and similar to our example and explanation here: https://yoast.com/wordpress-robots-txt-example/

    Plugin Support Jeroen Rotty

    (@jeroenrotty)

    I marked this thread as resolved due to a lack of activity by the original poster. The original poster is always welcome to re-open this topic.

    If you are not the original poster but have a similar issue, please read this post before opening a new request.

Viewing 7 replies - 1 through 7 (of 7 total)
  • You must be logged in to reply to this topic.