site overwhlemed (shut-down) by spiders and feeders
I have four sites, each with their own installed code base, on one hosting account. My sites go down, all at the same time, from time to time. This last time my host was able to tell me “your site was aggressively retrieved by several indexing/searching sites at one time, Yahoo! DE Slurp, Yahoo! Slurp, msnbot, msnbot-media, Sphere Scout, Feedfetcher-Google, Baiduspider+, Moreoverbot, ScoutJet, Googlebot”
I would restart the server and it would immediately go down again due to a limit of 15 consecutive connection limit to mysql.
I see my traffic using google analytics and do not have much traffic at all so I am at a loss about how to prevent this or solve this problem.
I don’t know which pages they are looking at/indexing.
I read somewhere that you can offload your rss feed to feedburner but I don’t know if that would solve this issue or not.
Any ideas are much appreciated!
- The topic ‘site overwhlemed (shut-down) by spiders and feeders’ is closed to new replies.