WordPress.org

1. msmeritt
   Member
   Posted 4 years ago #

   Just did a new WP installation. Initially set Options/Privacy so that search engines wouldn't see it. Used Google XML Sitemap plug-in and submitted it through Google Webmasters Tools just as I have done with 2 other WP installations. Naturally it didn't see the sitemap. Switched Options/Privacy, but Google still wouldn't see the sitemap and said I had a robots.txt issue.

   But I had no robots.txt file. Looked into it, created one. Added "Disallow: /" at first, on purpose, to prevent Google from seeing the site for a while during development. Then when I was ready, manually removed that line from robots.txt. So now the Privacy setting says that search engines should see the blog, robots.txt specifically allows all bots to crawl, but Google not only continues to tell me it can't access the site, it even believes that my robots.txt still has the "Disallow: /" line, which it absolutely doesn't.

   This leads me to believe that none of this has to do with robots.txt at all, at least not the one I created, since it seems like from the start there has been some issue.

   This was the first time I ever set Privacy to prevent search engines from seeing the site. It seems like there has been no recovery from that setting even though I have switched the setting back to let search engines see the site.

   Any thoughts at all?

2. msmeritt
   Member
   Posted 4 years ago #

   It looks like odds are good that this situation is working itself out. Google checks robots.txt only about once a day, so it took until today for it to see that I'd removed the "Disallow: /" from it. It now shows a verified sitemap and, though one little thing it hanging, I think it's going to clear itself out.

   Of course, none of this explains why Google couldn't see the site initially after I'd removed the Options/Privacy restriction and before I created the robots.txt file, so that's still a curiosity. Not so urgent to figure that out, but in case it suggests a problem that may repeat, if anyone has thoughts, I'd love to hear it, as a preventive measure for future installations.

Topic Closed

This topic has been closed to new replies.