I ask this because withing a day of me installing wordpress on my home system, it was hit by googlebot. I installed this on a private system, private address, that at the time had never been crawled (that I know of). After doing a second install Sunday, into a second directory, that was crawled on monday. I have many subdirectories on my server, all protected by anonymity. When you go to my site, there is just the Fedora Core test page. There are no links to browse to other directories on the server. Yet BOTH of my wordpress installs have been crawled, and NOTHING else on my server. I understand that I can use a robots.txt file to correct this, assuming the crawlers obey.
- The topic ‘Does wordpress notify crawlers and robots of the sites installation?’ is closed to new replies.