Moderator
Jan Dembowski
(@jdembowski)
Forum Moderator and Brute Squad
What if you had test or deployment installation that you don’t want indexed? That’s what that option is for, for those scenarios where you wish to tell the search engines to go away.
If you enable that option then WordPress will produce this for robot.txt requests.
User-agent: *
Disallow: /
Which just tells search engines that you don’t want your site crawled. It’s up to the search engines to honor that. That’s why the option is “Search engines discouraged” and not blocked.
Thread Starter
matre
(@matre)
I appreciate it. Thank you Dan.
Moderator
Jan Dembowski
(@jdembowski)
Forum Moderator and Brute Squad
You’re welcome. It’s “Jan” BTW. 😉