I've been using WordPress 2.0.4 on my server for a while now (hosted through a private web service provider). I recently modified the code for the index.php file to log all IP addresses and user-agents to a new table in a SQL database on my server.
I was surprised to find that immediately after I make a post to my blog (within 2 minutes after I post), Yahoo's RSS feed reader visits my index page. Shortly after Yahoo's server hits my site, search engine "bots" from other companies also hit the site. My site only gets hit by these search aggregators and by Yahoo immediately after I post. If I don't post for 3 or 4 days, they don't visit.
In my opinion, it seems like somehow my site is telling Yahoo that I've posted and it's time to come visit to get the latest post. I don't want this to happen - in fact, I don't want Yahoo or any other engine to visit my site at all! How do I turn off this "feature?" I've looked through the code, and from what little I know of PHP, I can't figure out what is making this happen. Any ideas about what is going on? I've already tried using a "disallow all" robots.txt file on my site and it hasn't kept these search engines from visiting me. Thanks for your help.