Crawler Server Load Limit on shared CloudLinux

  • goldensw

    (@goldensw)


    6 days, 16 hours ago

    Hi, I’m running LSCache WordPress plugin on shared hosting with LiteSpeed Enterprise and CloudLinux/CageFS isolation . The crawler works correctly, but I noticed that the Server Load Limit setting evaluates the global server load average, not the per-account load within my LVE.

    From my understanding, this makes the setting effectively unusable in shared CloudLinux environments:

    • On shared CloudLinux, the global load is dominated by other tenants’ activity, which I have no visibility into or control over.
    • A conservative value would block my crawler permanently (global baseline often exceeds it).
    • A high value effectively disables the protection and the setting becomes cosmetic.
    • CloudLinux LVE limits already throttle my account independently, so I can’t actually overload the server from my side regardless of crawler activity.

    Questions:

    1. Is there a way to make Server Load Limit evaluate per-account resource usage (CPU, EP, IOPS within the LVE) instead of global server load average?
    2. Is there a server-side variable, filter, or wp-config.php constant that changes how the threshold is interpreted on CloudLinux?
    3. If no such option exists, is this a known limitation, and is there any roadmap consideration for shared/CloudLinux-aware load detection?

    For context: my hosting plan has dedicated LVE limits (CPU, EP, IOPS, memory), and CloudLinux automatically throttles overuse. The crawler workload is small (~280 URLs, ~70s per cycle) and well within my plan’s capacity, but the load limit setting offers no meaningful control in this environment.

    Any guidance is appreciated. Thank you!

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Support qtwrk

    (@qtwrk)

    6 days, 13 hours ago

    for 1) no , that’s how Linux system works , when you get load, you get load of entire server , this can not be separated account-basis

    for 2/3) you can set per-vhost level SetEnv CRAWLER_LOAD_LIMIT 1 directive to set different load level restriction , like vhost A set to 5 , vhost B set to 10 , vhost C set to 15 …etc , but this has to be done by the sys admin

    it’s not exactly the way you want, but due how the way how Linux system works , there is no other way around , best thing you can do is set different load restriction for different level of vhost , like some has more resources gets more allowance , some has less gets less ….etc

    Plugin Support litetim

    (@litetim)

    6 days, 10 hours ago

    @goldensw
    Let’s test this:
    I have added a test code to this branch: https://github.com/timotei-litespeed/lscache_wp/commit/080e5e1e8b544926a86cf98eb4d8a89331921468
    You can test the code from LSC => Toolbox => Beta test Paste the link above and click on Update.

    THIS is a dev build, please test it on an enviroment that is not sensitive.
    Please tell me if that is working.

    • This reply was modified 6 days, 10 hours ago by litetim.
    • This reply was modified 6 days, 10 hours ago by litetim.
    Thread Starter goldensw

    (@goldensw)

    6 days, 10 hours ago

    Thank you for the response. One follow-up thought: cPanel already reads per-account CPU/memory/IOPS usage in real-time from CloudLinux LVE and displays it in Resource Usage. Would it be feasible in any way for the crawler to use those same per-account metrics instead of global load average? The data is already there, it’s just not being used by the crawler.

    Plugin Support litetim

    (@litetim)

    6 days, 10 hours ago

    @goldensw I think the difference stands in how CPANEL and PHP reads the CPU usage.
    We are using: https://www.php.net/manual/en/function.sys-getloadavg.php to read current load.

    The code will try to look for usage based on server user ID. Please give it a try: https://github.com/timotei-litespeed/lscache_wp/commit/080e5e1e8b544926a86cf98eb4d8a89331921468

    Thread Starter goldensw

    (@goldensw)

    6 days, 9 hours ago

    Thank you!

    I see your code checks is_readable('/proc/lve/list') but on this server it returns false:

    php -r "var_dump(is_readable('/proc/lve/list'));"
bool(false)

    No LVE debug output appeared in debug.log after running the crawler manually if that helps.
    Perhaps I could help you with more testing if you tell me what would help if you think this is worth exploring further.

    Plugin Support qtwrk

    (@qtwrk)

    6 days, 9 hours ago

    you are right , that wouldn’t work , unless a php user has a way to extract this info, otherwise I don’t think this would work

    Thread Starter goldensw

    (@goldensw)

    6 days, 8 hours ago

    Thank you for trying! If you get any more ideas do let me know and I will be happy to test. 🙂

    Plugin Support qtwrk

    (@qtwrk)

    6 days, 4 hours ago

    I have spent some time wandering around to explore the possibilities , but it seems there is no way to do it by PHP , shared hosting are meant to be restricted and tightened for security concern , as result, many functions , files , access , are denied by purpose

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.