The Support Forums will be in read-only mode for a scheduled maintenance window on 01 September 2016 14:00 UTC - 20:00 UTC. More information.

Robots + Maintinence + Cache Refresh = Crash (1 post)

  1. Norbert McDorbert
    Posted 2 years ago #


    I have a dedicated server, 2.6 qc / Sata hdd (yes, i know) / 8gb Ram

    LAMP setup box

    On it contains 1 WordPress multisite with appx 200 sites, 100 active.

    During the day, its fine. Sure some times dashboard takes an extra second, but its fine. Logged in users have 3-4 sec page loads; Cached pages are being served in 1-2 seconds total; all is well.

    Come 1am; my Cpanel / WHM scripts fire off for Nightly Backups. Bots and Crawlers come out of the woodwork; and my SuperCache refreshes.

    = Crash.

    So many robots crawling all sorts of inane pages, that my server becomes non responsive.

    Cloudflare was suggested but that is not a good option as my site is Domain Mapped

    Varnish was NOT recommended by host as they think it will cause issues down the road.

    Their only answer was to Drop connections; however I worry about the SEO impact that has on primary site.

    What do you all Suggest

    (Using WP SUPER CACHE - Preload is kind of f'd as it deletes before refreshes vs overwrites; so if it breaks it breaks.

Topic Closed

This topic has been closed to new replies.

About this Topic

  • RSS feed for this topic
  • Started 2 years ago by Norbert McDorbert
  • This topic is not resolved
  • WordPress version: 3.6.1