Crawler stopped_highload

  • Resolved gecko002002

    (@gecko002002)


    1 year, 8 months ago

    Hello guys,

    So my crawler stopped working for a few days, actually it is stopping after crawling a few pages, like 30 out of more than 1000. Ended reason: stopped_highload.

    I tried to restart the crawler manually or reset position but nothing happens.

    Also I have a lot of cron jobs defined, 8 of them. Are all of them necessarily? Or can I disable some of them and run the crawler more often?

    Cron jobs: https://ibb.co/wRG30wj
    Litespeed settings: https://ibb.co/6JYNF0y

    For hosting I have something like this: WordPress PRO 3 SSD – 200 GB storage SSD, 16 GB RAM, 6 x 3,2 Ghz CPU

    Thanks guys for any help!

    The page I need help with: [log in to see the link]

Viewing 15 replies - 1 through 15 (of 23 total)
1 2
  • Plugin Support Hai Zheng⚡

    (@hailite)

    1 year, 8 months ago

    You can try a higher number for Server Load Limit setting.

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    @gecko002002

    If you have SSH access, run “uptime” command to see the average load of your server. If done please post these 3 values.

    Are you on shared hosting or is your server dedicated?

    Thread Starter gecko002002

    (@gecko002002)

    1 year, 8 months ago

    @hailite this did the trick, the stopped_highload error dissapeared. It ran for a while but now I get the error stopped_maxtime.

    @serpentdriver unfortunately its a shared server and I dont have access to SSH.

    Thread Starter gecko002002

    (@gecko002002)

    1 year, 8 months ago

    @hailite actually I was wrong, I got the stopped_highload again.

    Plugin Support Hai Zheng⚡

    (@hailite)

    1 year, 8 months ago

    Try v5.2 (latest dev)

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    @gecko002002

    @serpentdriver unfortunately its a shared server and I dont have access to SSH.

    No problem. There are other methods to find the best settings so that the crawler does not terminate prematurely due to excessive load or timeout. However, the possibilities are limited on shared hosting, so you first have to find out what the ideal value for Server Load Limit is. The Server Load Limit setting is not very helpful for this. Especially since the value for the load limit is different on each server. This is especially true for shared hosting. But I can help you to find the right settings. But I need your support for this.

    Shall we do this together?

    Thread Starter gecko002002

    (@gecko002002)

    1 year, 8 months ago

    Yes of course! What do you need from me?

    But now it seems to work, maybe because its weekend and the server is less stressed I guess.

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    There is not much you have to do. To get the information we need you must install PHPSysInfo, a script that displays information about your system nicely on your server. Install is easy. Just download it, create a directory (wherever you want) with name of your choice on your server and upload the script. In the root directory of this script there is a file named phpsysinfo.ini.new. Rename it to phpsysinfo.ini, that’s it.

    There is a demo available that you can find at: http://phpsysinfo.sourceforge.net/multi/index.php?disp=bootstrap

    Download from Sourceforge: https://phpsysinfo.github.io/phpsysinfo/

    Request the script in your browser.

    Thread Starter gecko002002

    (@gecko002002)

    1 year, 8 months ago

    Hello,

    Sorry for the late response, had a few busy days.
    Here is the script that you told me about.

    https://craftup.ro/system/index.php

    We did what you said but it doesnt seem to work.

    Thanks!

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    @gecko002002

    Did you rename phpsysinfo.ini.new into phpsysinfo.ini?

    Thread Starter gecko002002

    (@gecko002002)

    1 year, 8 months ago

    Yes, we did.

    https://ibb.co/ZR93mDQ

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    Okay, then I have bad news for you. Your provider removed all files and functions on your server to get information about the load. This also means that LiteSpeed cache plugin also can’t use load information because it also needs these system files.

    If you click on the warning shield on top of the page you will see what is missing to run PhpSysInfo.

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    Did you already give up? 😉 We have one last chance. Are you still ready? 🙂

    Thread Starter gecko002002

    (@gecko002002)

    1 year, 8 months ago

    Haha, of course not. But this could also be a problem with our website. Some plugin or something that uses a lot of CPU.

    Also was thinking about contacting the hosting provider to ask them about phpsysinfo

    But if you have any ideeas, I cant wait to hear them 🙂

    Thanks again!

    serpentdriver

    (@serpentdriver)

    1 year, 8 months ago

    Okay, just create a blank php file with a name of your choice and place it wherever you want. Paste the code below into this file and request it in browser. I have no hope that it works, but hope dies last…. 🙂

    <?php
$load = sys_getloadavg();
echo $load[0];
Viewing 15 replies - 1 through 15 (of 23 total)
1 2
  • The topic ‘Crawler stopped_highload’ is closed to new replies.