I have checked the box under the privacy settings section that says "I would like to block search engines, but allow normal visitors". But I noticed today that my blog shows up in the search results for Google. How can this be? I don't understand. I just have it on maintenance mode, but I wanted to keep it hidden until I finish setting it up. Does this box pertain to wordpress SUBMITTING my blog to search engines---but crawlers would find it anyway. Is that what's up?
Why does my blog show in Google search---I checked box to block search engines? (3 posts)
-
leonpyoder
Posted 3 years ago # -
You can use a robots.txt file to request that search engines remove your site and prevent robots from crawling it in the future. (It's important to note that if a robot discovers your site by other means - for example, by following a link to your URL from another site - your content may still appear in our index and our search results. To entirely prevent a page from being added to the Google index even if other sites link to it, use a noindex meta tag.)
To prevent robots from crawling your site, place the following robots.txt file in your server root:
User-agent: *
Disallow: /To remove your site from Google only and prevent just Googlebot from crawling your site in the future, place the following robots.txt file in your server root:
User-agent: Googlebot
Disallow: /Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below.
For your http protocol (http://yourserver.com/robots.txt):
User-agent: *
Allow: /For the https protocol (https://yourserver.com/robots.txt):
User-agent: *
Disallow: /
Topic Closed
This topic has been closed to new replies.
