Excessive Spider crawl after using All-in-One Event Calendar

  • Resolved Ian Parkinson

    (@ianpark)


    10 years, 4 months ago

    Hi guys. Tearing my hair out now. So, made the mistake of using All-in-One Event Calendar by Timely.
    After experiencing massive cpu usage I found the fix that they had on this page limiting excessive crawls
    That was back in WP version 3.6 and it worked. when 3.7 came out all the problems erupted again and then multiplied, as follows.
    1. the fix now longer worked.
    2. I uninstalled the plugin, but it left thousands of lines of code in the database, and the crawling continued.
    3. I deleted every page or post that had anything to do with that calendar, and deleted every reference I could find to it in the database but the crawling continued.
    4. Much of the crawl was from google, but much of it was from elsewhere. I blocked out google by discouraging search engines from indexing the site for a few days. The excessive crawling continued from many other places however.
    5. I changed the directory in which WordPress resides in the hope that this would remove the site from the targeting of this crawl but to no avail.
    I cannot understand why many spiders (lots from China IP 180.76.6.xxx etc) keep hitting my site, crawling pages like
    calendar-2/action~agenda/page_offset~-1/time_limit~1385251199/cat_ids~14,16,168,5
    and
    ai1ec_event/sunday-service-munno-para/?instance_id=933 HTTP/1.1″ 403 917 “-“
    These events have long been deleted from everything on the site.
    6. So I used iQ Block Country plugin to block China, but I’m still getting those crawls from China plus others from elsewhere, mostly crawling relics from all-in-one event calendar that are no longer on the site.
    7. HELP! … I think I’m at my wits end, and out of ideas. Can anyone else help me?
    Site looks fine from the front end, just keep getting warnings from my host that I’ll be shut down if that cpu usage doesn’t go down.
    So site is http://www.lifechurchsa.org.au/
    Thanks in advance 🙂

Viewing 3 replies - 1 through 3 (of 3 total)
  • Pioneer Web Design

    (@swansonphotos)

    10 years, 4 months ago

    Have you checked your .htaccess file? Manual Update? Renamed the whole plugins folder to plugins.old. renamed current theme to theme_folder_name.old.

    Can you write a redirect or rewrite rule that covers the majority of the ‘calendar’ links?

    All of the above done, you cannot control who or what hits your site. If you feel that your site is being attacked, either tell your host to block the attacks or find a host who can and will.

    Pascal

    (@iqpascal)

    10 years, 4 months ago

    Hi,

    The iQ Block Country plugin only blocks wordpress pages/blogs so no images for instance. However they do show up in the logfiles and some statistics plugins as the url was accessed, it was just blocked with a 403 and go away message.

    Does this mean a bot will not access the same page again? Nah. You will keep on seeing hits to those posts/files even if they no longer exist because you deleted them for instance. This may go on for years.

    Automatic bots crawl your site looking for bad versions of phpmyadmin, joomla, wordpress plugins etc just so they can see if they can enter your site somewhere and use it for malware for instance.

    You just have to remember that they are just crawls. It’s like a burglar scouting your home, it may even try to look if your frontdoor is left open. But as long as you keep good care of your house that’s probably all they do.

    If all they do is crawl I wouldn’t worry about it too much.

    Thread Starter Ian Parkinson

    (@ianpark)

    10 years, 4 months ago

    Thanks for your response guys. Hey Pioneer, there’s a couple of good ideas. I’ll implement those and report back.
    @pascal:

    If all they do is crawl I wouldn’t worry about it too much.

    If my host will agree to that I will too. After all, it’s my host who are bugging me about cpu usage 😉

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Excessive Spider crawl after using All-in-One Event Calendar’ is closed to new replies.