LiteSpeed Cache Crawler hangs and never completes

  • Sipo

    (@bockeromislam)


    4 days, 6 hours ago

    Hello,

    We’re experiencing two crawler issues on LiteSpeed Cache v7.8.1 running on OpenLiteSpeed (VPS).

    1. Sometimes the crawler stops with the reason stopped_reset.
    2. Other times it starts normally, crawls around 280 URLs, then stops making progress and never resumes, even though it still appears to be running.

    We have already verified:

    • Server IP is correct.
    • XML sitemap is accessible.
    • Product pages are cacheable.
    • Blocklist is empty.
    • No timeout issues (TTFB ~0.04s).
    • Server load is normal.
    • WP-Cron is enabled.
    • No special crawler-related constants are defined in wp-config.php.

    For testing, we also changed the Crawler Run Interval to 60 seconds, but the crawler still does not continue or restart after stopping.

    This is a VPS, and we also have a real server cron running every 5 minutes:

    */5 * * * * wget -q -O - https://example.com/wp-cron.php?doing_wp_cron >/dev/null 2>&1

    We also tested:

    curl -I https://example.com/wp-cron.php
    wp cron event run --due-now --allow-root

    without resolving the issue.

    Is this a known crawler issue, or is there any additional debug logging you recommend enabling?

    Thank you.

Viewing 13 replies - 1 through 13 (of 13 total)
  • Plugin Support qtwrk

    (@qtwrk)

    4 days, 5 hours ago

    60 seconds probably bad idea and make crawler confuse , make it at least 300

    but please try enable debug log over debug include URI async_litespeed , then try to run crawler wth WP CLI or your wget , see what logs says

    Thread Starter Sipo

    (@bockeromislam)

    4 days, 3 hours ago

    I increased the crawler interval from 60 seconds to 300 seconds as suggested.

    Enabled the Debug Log and added async_litespeed to the Debug Include URI.

    I then reset the crawler position and tested it again by running it manually, through my server cron. I also restarted OpenLiteSpeed.

    At first, the crawler stopped immediately with stopped_reset. After restarting OpenLiteSpeed and resetting the crawler again, it started working.

    According to the debug log, the crawler starts successfully, takes over the crawler lane, passes the local HTTPS port test, resolves DNS correctly, and begins crawling URLs normally. It also updates the crawler position continuously without any PHP errors or timeout messages.

    However, after running for approximately 5 minutes, it stops making progress. The crawler does not complete, and no obvious error is logged. Sometimes it later resumes by itself, while other times it remains stuck until I manually trigger it again.

    Can i send the debug log privately?

    • This reply was modified 4 days, 3 hours ago by Sipo.
    Plugin Support qtwrk

    (@qtwrk)

    4 days, 3 hours ago

    you can bulk-replace the IP/domain or any other info you see private in the log , then share it over https://pastebin.com/

    Thread Starter Sipo

    (@bockeromislam)

    4 days, 2 hours ago

    Wouldnt let me post whole debug but i’ve shorten it, let me know if you need anything else:

    https://pastebin.com/yJtCZkWa

    • This reply was modified 4 days, 2 hours ago by Sipo.
    Plugin Support qtwrk

    (@qtwrk)

    4 days, 2 hours ago

    this shows the crawling start part

    what did crawler summary page shows when it stops ? stop reset ?

    Thread Starter Sipo

    (@bockeromislam)

    4 days, 2 hours ago

    Initially it showed stopped_reset.

    After resetting the crawler and restarting OpenLiteSpeed, the behavior changed.

    The crawler now starts normally and the summary page continuously updates the position for about 4–5 minutes.

    After that, the position stops updating and the crawler appears to hang. It does not complete the crawl or continue to the next position.

    The status remains crawling, updated position (the last recorded update), but no further progress is made.

    https://ibb.co/rK01hx2K

    LiteSpeed Icon Start watching… 06 Jul 2026 21:23:22 Size: 1873 Crawler: #1 Position: 1 Threads: 3 Status: stopped_reset 06 Jul 2026 21:23:22 Size: 1873 Crawler: #1 Position: 1 Threads: 3 Status: crawling, prepare running 06 Jul 2026 21:27:05 Size: 1873 Crawler: #1 Position: 16 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:27:16 Size: 1873 Crawler: #1 Position: 28 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:27:28 Size: 1873 Crawler: #1 Position: 40 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:27:40 Size: 1873 Crawler: #1 Position: 52 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:27:53 Size: 1873 Crawler: #1 Position: 64 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:28:06 Size: 1873 Crawler: #1 Position: 76 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:28:17 Size: 1873 Crawler: #1 Position: 85 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:28:29 Size: 1873 Crawler: #1 Position: 97 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:28:42 Size: 1873 Crawler: #1 Position: 109 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:28:56 Size: 1873 Crawler: #1 Position: 121 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:29:08 Size: 1873 Crawler: #1 Position: 133 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:29:21 Size: 1873 Crawler: #1 Position: 145 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:29:35 Size: 1873 Crawler: #1 Position: 157 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:29:47 Size: 1873 Crawler: #1 Position: 169 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:29:59 Size: 1873 Crawler: #1 Position: 181 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:30:13 Size: 1873 Crawler: #1 Position: 193 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:30:25 Size: 1873 Crawler: #1 Position: 205 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:30:38 Size: 1873 Crawler: #1 Position: 217 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:30:51 Size: 1873 Crawler: #1 Position: 229 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:31:04 Size: 1873 Crawler: #1 Position: 241 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:31:18 Size: 1873 Crawler: #1 Position: 253 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:31:31 Size: 1873 Crawler: #1 Position: 265 Threads: 3 Status: crawling, updated position 06 Jul 2026 21:31:44 Size: 1873 Crawler: #1 Position: 277 Threads: 3 Status: crawling, updated position ………………….

    Thread Starter Sipo

    (@bockeromislam)

    4 days, 2 hours ago

    I have another observation.

    The crawler resumed automatically after being idle for about one hour. The debug log shows:

    Lane file is older than 1h, releasing lane
    Release lane
    Take over lane as lane is free
    ......crawler started......

    It then continued from position 277 (it did not start over) and crawled normally until position 556.

    After approximately another 4–5 minutes, it stopped making progress again and no further position updates appeared.

    This behavior seems to repeat: the crawler runs normally for a few minutes, then stops, and later resumes only after the lane is considered older than one hour.

    • This reply was modified 4 days, 2 hours ago by Sipo.
    Plugin Support qtwrk

    (@qtwrk)

    4 days, 2 hours ago

    please try this , set log size to 30MB , debug URI to both async_litespeed and wp-cron.php , clean up existing , then run your curl -XGET -I .....wp-cron.php every 5 minutes , see what did full log shows in 30 or 60 minutes period.

    Thread Starter Sipo

    (@bockeromislam)

    4 days, 1 hour ago

    Did what you asked for. Here’s the debug log after approximately 30 minutes:

    https://pastebin.com/D9RcQNrH

    Password: r7kikQqfxT

    • This reply was modified 4 days, 1 hour ago by Sipo.
    Thread Starter Sipo

    (@bockeromislam)

    3 days, 2 hours ago

    It now looks like it has crawled and cached majority of my pages.

    Out of 1872 pages
    1852 green
    21 blue

    However, when i browse my products with console alot of them is still (x-litespeed-cache – miss). They are not shown as blocked or blue. Im stuck, how can we proceed? Can the issue regarding lane file be something to look into?

    • This reply was modified 3 days, 2 hours ago by Sipo.
    • This reply was modified 3 days, 2 hours ago by Sipo.
    Plugin Support qtwrk

    (@qtwrk)

    3 days ago

    on your log, around line 191 and 1235, there are 2 purges takes place, which will stop crawler and purge cache

    these 2 are from cron job , YITH_WC_Recover_Abandoned_Cart_Helper->clear_coupons , I think it’s related to this abandoned cart plugin

    Thread Starter Sipo

    (@bockeromislam)

    2 days, 20 hours ago

    I disabled the YITH Recover Abandoned Cart plugin, and the purge events are no longer present in the debug log. However, the crawler still stops after about 4–5 minutes.

    After that, it appears to freeze for around 1–2 hours. During that time I cannot get it to continue, whether I start it manually from the LiteSpeed Cache admin page or trigger it via SSH (wp-cron).

    Eventually it resumes by itself, crawls for another 4–5 minutes, and then freezes again. This behavior repeats consistently.

    https://gist.github.com/Sipoo/bdc679c6d7e56053c55b54a087bda96c

    • This reply was modified 2 days, 20 hours ago by Sipo.
    Plugin Support qtwrk

    (@qtwrk)

    1 day ago

    the log only contains like 6 minutes

    the crawler runs for 5 mintues, than 10 minutes reset , then it should start again

    please try this

    launch the crawler, by whatever method you see fit , by admin page or wp-cron or wp cli , mark the down

    5 minutes later , run or make sure the wp-cron is triggered

    then another 5 minutes later , run or make sure wp-cron is triggered

    wait for another 5 , run wp-cron again

    now it should be over 20 minutes , share the log , see what it shows

    by the way , make sure you have set enable crawler to on in crawler setting.

Viewing 13 replies - 1 through 13 (of 13 total)

You must be logged in to reply to this topic.

  • 13 replies
  • 2 participants
  • Last reply from: qtwrk
  • Last activity: 1 day ago
  • Status: not resolved