Robots.txt not disallowing based on “Search engine visibility” setting

  • Resolved pauljsilver

    (@pauljsilver)


    10 months, 3 weeks ago

    Hi, I’ve got a website that’s not ready to be live yet, but is on a public address. I’ve got the “Search engine visibility” setting in WordPress ticked to “Discourage search engines from indexing this site” (in Settings > Reading) which up until recently has meant the robots.txt has this entry in it:

    User-agent: *
    Disallow: /

    But having had someone find the site through search, I’ve discovered currently robots.txt is set as this:

    START YOAST BLOCK —————————
    User-agent: *
    Disallow:
    (Taking out the sitemap line to keep the site private)

    Even though the WP setting is definitely still on “discourage.” I looked up how to set it within Yoast, and their help page says to set the WP setting that I have done.

    Does anyone know if there’s a new place to set this? I’m using the latest Yoast, just triggered the update this morning. I could set it in code as well if that’s the only way to get it working.

    Thanks

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    10 months, 3 weeks ago

    Hello @pauljsilver,

    Thanks for using Yoast SEO. When you install WordPress without having Yoast SEO, WordPress generates a virtual robots.txt file at /?robots=1. Once you install Yoast SEO, we hook into the dynamic file, and you can use our file editor tool to create a static robots.txt file at /robots.txt.

    There have been instances where there might be a slight difference in the contents of /?robots=1 VS /robots.txt. So, it’s in our product roadmap to revisit our robots.txt implementation. 

    In the meantime, you can add the disallow directive directly in the file editor of Yoast SEO. Note that though you discourage search engines, it’s up to them to honor your directives.

    Thread Starter pauljsilver

    (@pauljsilver)

    10 months, 2 weeks ago

    Thank you @maybellyne, I’ve used the tools section and made the robots.txt file.

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Robots.txt not disallowing based on “Search engine visibility” setting’ is closed to new replies.