Title: Crawler works randomly Last modified: October 28, 2025 --- # Crawler works randomly * Resolved [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/) * Hi, * I am currently setting up LS Cache on some of our websites hosted at 02witch, a provider who supplies LiteSpeed Cache functionalities. * I can set up LS Cache with all the functionalities that I need (including Memcached), but when it turns to setting up the crawler (compulsory to have a preloaded cache), the only thing I can say is that sometimes it works, sometimes it doesn’t. * I have exactly the same LS Cache plugin setup as well as hosting parameters ( PHP versions and modules,….). * For instance, crawler works with: * [https://www.conciergerie-annecy.fr/](https://www.conciergerie-annecy.fr/) * And does not with: * [https://mapetitecom.fr/](https://mapetitecom.fr/) * On the latest website, clicking on the Manual Execution button just leads to… nothing. * Crawler status is (in french) : Démarrage de l’observation. * Of course, my XML sitemap is correctly setup and discovered. * I really like the performance brought by LS Cache but I need to have a reliable crawler functionality to get rid of WP Rocket, which has become much to expensive. * Cheers. * The page I need help with: _[[log in](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fwordpress.org%2Fsupport%2Ftopic%2Fcrawler-works-randomly%2F%3Foutput_format%3Dmd&locale=en_US) to see the link]_ Viewing 15 replies - 1 through 15 (of 27 total) 1 [2](https://wordpress.org/support/topic/crawler-works-randomly/page/2/?output_format=md) [→](https://wordpress.org/support/topic/crawler-works-randomly/page/2/?output_format=md) * Plugin Support [litetim](https://wordpress.org/support/users/litetim/) * (@litetim) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18698473) * [@misterpo](https://wordpress.org/support/users/misterpo/) please create a report and share the ID generated. You can generate the report from LSC => Toolbox = > Report => click on “send to Litespeed” and share the generated ID.Also, please enable debug in LSC([here](https://docs.litespeedtech.com/lscache/lscwp/toolbox/#debug-log)) and look into Crawler Logs for the logs that are added.Please share the logs( toolbox => log view) on sites like: [https://pastebin.ubuntu.com/](https://pastebin.ubuntu.com/) - This reply was modified 7 months, 3 weeks ago by [litetim](https://wordpress.org/support/users/litetim/). * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18698498) * Hi, * Thx for your feedback. * Report number is: **DCCVZPKW** * Logs: [https://pastebin.ubuntu.com/p/ChYsfygf5p/](https://pastebin.ubuntu.com/p/ChYsfygf5p/) * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18698979) * the log is not showing complete message * please try add `async_litespeed` into debug include URI and try to grab again * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18699278) * Hi, * Done. * Here is the new debug log : [https://pastebin.ubuntu.com/p/WgNtQyQtwq/](https://pastebin.ubuntu.com/p/WgNtQyQtwq/) * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18700551) * the log doesn’t look alright * please confirm you have put this * ![](https://i0.wp.com/i.imgur.com/4gUHL1i.png?ssl=1) * and just in case you may need to clean up existing log first in order to get a clean log status * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18700904) * Hi, * The mentioned setting is correct. * Please find the link to the new log file : [https://pastebin.ubuntu.com/p/3R6WZR4JNd/](https://pastebin.ubuntu.com/p/3R6WZR4JNd/) * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18701959) * yeah , it looks like the request to start the crawler never reached your site * ```wp-block-code 100, 'blocking' => true, 'sslverify' => false, // 'cookies' => $_COOKIE,);$url = 'https://' . $_SERVER['HTTP_HOST'] . '/wp-admin/admin-ajax.php?action=async_litespeed&nonce=123123123&litespeed_type=crawler';$result = wp_remote_post(esc_url_raw($url), $args);echo '
';var_dump($result);echo '
'; ``` * please create a php file, name it like `test.php` with above code , put it at your worpdress root directory * then open it as `https://domain.com/test.php` , in private window , see what it returns * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18702188) * Hi, * I have create a phpinfo.php file which returns the expected output: * [https://mapetitecom.fr/phpinfo.php](https://mapetitecom.fr/phpinfo.php) * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18702198) * Sorry, I have created the test.php file which leads to a never ending loop * [https://mapetitecom.fr/test.php](https://mapetitecom.fr/test.php) * I have temporarily disabled preloader, please find below the output: * array(6) { [“headers”]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)# 2563 (1) { [“data”:protected]=> array(9) { [“date”]=> string(29) “Fri, 31 Oct 2025 07:54:12 GMT” [“content-type”]=> string(24) “text/html; charset=UTF-8” [“ vary”]=> string(15) “Accept-Encoding” [“x-dns-prefetch-control”]=> string(2) “ on” [“link”]=> string(208) “; rel=”[https://api.w.org/”](https://api.w.org/”);,; rel=”alternate”; title=”JSON”; type=”application/json”, ; rel=shortlink” [“x- litespeed-cache-control”]=> string(8) “no-cache” [“cache-control”]=> string(55)“ no-cache, must-revalidate, max-age=0, no-store, private” [“server”]=> string( 22) “o2switch-PowerBoost-v3” [“content-encoding”]=> string(4) “gzip” } } [“body”] => string(188321) “ * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18702429) * yeah , that request is supposedly to return “0” , but in your case * ```wp-block-code ["body"]=> string(188771) " ``` * it’s hitting a html page * which means something blocked or redirected this request, you need to undo that first * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18702458) * I have the following code in .htaccess file. Is it expected ? * marker ASYNC start * RewriteCond %{REQUEST_URI} /wp-admin/admin-ajax.php RewriteCond %{QUERY_STRING} action=async_litespeedRewriteRule .* – [E=noabort:1] marker ASYNC end * Plugin Support [qtwrk](https://wordpress.org/support/users/qtwrk/) * (@qtwrk) * [7 months, 3 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18702466) * yes , this is expected, something else blocked or redirected * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 2 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18706522) * Hi, * I finally worked it out: crawler does not work on websites with Imagify plugin installed in rewrite rule mode. * Here is the set of .htaccess rules added by Imagify: * BEGIN Imagify: webp file type * AddType image/webp .webp END Imagify: webp file type BEGIN Imagify: avif file type * AddType image/avif .avif END Imagify: avif file type BEGIN Imagify: rewrite rules for avif * # Vary: Accept for all the requests to jpeg, png, and gif. SetEnvIf Request_URI“.( jpg|jpeg|jpe|png|gif|webp)$” REQUEST_image * RewriteEngine OnRewriteBase / * ```wp-block-code # Check if browser supports AVIF images. # Update the MIME type accordingly. RewriteCond %{HTTP_ACCEPT} image/avif # Check if AVIF replacement image exists. RewriteCond %{REQUEST_FILENAME}.avif -f # Serve AVIF image instead. RewriteRule (.+)\.(jpg|jpeg|jpe|png|gif|webp)$ $1.$2.avif [T=image/avif,NC] ``` * # Update the MIME type accordingly. Header append Vary Accept env=REQUEST_image END Imagify: rewrite rules for avif BEGIN Imagify: rewrite rules for webp * # Vary: Accept for all the requests to jpeg, png, and gif. SetEnvIf Request_URI“.( jpg|jpeg|jpe|png|gif)$” REQUEST_image * RewriteEngine OnRewriteBase / * ```wp-block-code # Check if browser supports WebP images. RewriteCond %{HTTP_ACCEPT} image/webp # Check if WebP replacement image exists. RewriteCond %{REQUEST_FILENAME}.webp -f # Serve WebP image instead. RewriteRule (.+)\.(jpg|jpeg|jpe|png|gif)$ $1.$2.webp [T=image/webp,NC] ``` * Header append Vary Accept env=REQUEST_image END Imagify: rewrite rules for webp * Any idea of what affects /wp-admin/admin-ajax.php?action=async_litespeed&nonce =123123123&litespeed_type=crawler request? * Thread Starter [misterpo](https://wordpress.org/support/users/misterpo/) * (@misterpo) * [7 months, 2 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/#post-18706761) * I confirm your say regarding O2switch. * However, LS crawler doesn’t work only when Imagify is installed, I have made a benchmark of nearly 15 websites. * Plugin Support [litetim](https://wordpress.org/support/users/litetim/) * (@litetim) * [7 months, 2 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/page/2/#post-18707650) * [@misterpo](https://wordpress.org/support/users/misterpo/) can you make a test? disable Imagify and test the crawler a few days(at least until next run). See if this changes anything. Viewing 15 replies - 1 through 15 (of 27 total) 1 [2](https://wordpress.org/support/topic/crawler-works-randomly/page/2/?output_format=md) [→](https://wordpress.org/support/topic/crawler-works-randomly/page/2/?output_format=md) The topic ‘Crawler works randomly’ is closed to new replies. * ![](https://ps.w.org/litespeed-cache/assets/icon-256x256.png?rev=2554181) * [LiteSpeed Cache](https://wordpress.org/plugins/litespeed-cache/) * [Frequently Asked Questions](https://wordpress.org/plugins/litespeed-cache/#faq) * [Support Threads](https://wordpress.org/support/plugin/litespeed-cache/) * [Active Topics](https://wordpress.org/support/plugin/litespeed-cache/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/litespeed-cache/unresolved/) * [Reviews](https://wordpress.org/support/plugin/litespeed-cache/reviews/) * 27 replies * 4 participants * Last reply from: [litetim](https://wordpress.org/support/users/litetim/) * Last activity: [7 months, 2 weeks ago](https://wordpress.org/support/topic/crawler-works-randomly/page/2/#post-18709167) * Status: resolved