Robots Maintinence Cache Refresh = Crash

  • Norbert McDorbert

    (@dadreviewed)


    10 years, 7 months ago

    Hello

    I have a dedicated server, 2.6 qc / Sata hdd (yes, i know) / 8gb Ram

    LAMP setup box

    On it contains 1 WordPress multisite with appx 200 sites, 100 active.

    During the day, its fine. Sure some times dashboard takes an extra second, but its fine. Logged in users have 3-4 sec page loads; Cached pages are being served in 1-2 seconds total; all is well.

    Come 1am; my Cpanel / WHM scripts fire off for Nightly Backups. Bots and Crawlers come out of the woodwork; and my SuperCache refreshes.

    = Crash.

    So many robots crawling all sorts of inane pages, that my server becomes non responsive.

    Cloudflare was suggested but that is not a good option as my site is Domain Mapped

    Varnish was NOT recommended by host as they think it will cause issues down the road.

    Their only answer was to Drop connections; however I worry about the SEO impact that has on primary site.

    What do you all Suggest

    (Using WP SUPER CACHE – Preload is kind of f’d as it deletes before refreshes vs overwrites; so if it breaks it breaks.

  • The topic ‘Robots Maintinence Cache Refresh = Crash’ is closed to new replies.