Plugin Support
qtwrk
(@qtwrk)
is it possible that your cloudflare setting blocks the crawler request ?
How can I check this setting? Also, this problem started after the update. I didn’t make any changes to the Cloudflare settings. Did you add a separate feature for Cloudflare with the new update? Is there a setting we need to make?
Plugin Support
qtwrk
(@qtwrk)
okay , please go to toolbox -> debug setting -> set debug log to ON , debug level to advance
then put a smaller size sitemap , like only 50 – 100 URLs , crawl it , see how it shows , and then share the crawler log , you can find the log file in wp-content/litespeed/debug
03/28/25 18:58:05.373 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.375 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/06/refox-648c295cf170a.webp
03/28/25 18:58:05.375 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.375 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/06/refox-648c295cf170a-150×47.webp
03/28/25 18:58:05.375 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.375 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/06/refox-648c295cf170a-64×20.webp
03/28/25 18:58:05.375 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/07/potislogo.webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/07/potislogo-400×187.webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/07/potislogo-150×70.webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/07/potislogo-430×201.webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] already webp
03/28/25 18:58:05.376 [66.249.73.130:54856 1 D01] [Media] webp replacing: https://x/wp-content/uploads/2023/07/potislogo-64×30.webp
The debug file continues like this. note: I added an x instead of the site name
03/28/25 19:01:39.976 [103.204.128.131:37396 1 z6a] 🕸️ [status] 😅 Blacklisted [url] https://x/produkt/shelving-system-for-cooling-units-940mm/
03/28/25 19:01:39.976 [103.204.128.131:37396 1 z6a] 🕸️ [status] 😅 Blacklisted [url] https://x/produkt/heavy-duty-saute-pans-50l-electric-2/
03/28/25 19:01:39.976 [103.204.128.131:37396 1 z6a] 🕸️ [status] 😅 Blacklisted [url] https://x/produkt/crepeplade/
Sorry, I guess I re-scanned the ones that were already cached in my first reply. The list continues as below in my second reply.
03/28/25 19:01:39.976 [103.204.128.131:37396 1 z6a] 🕸️ [status] 😅 Blacklisted [url] https://x/produkt/shelving-system-for-cooling-units-940mm/
03/28/25 19:01:39.976 [103.204.128.131:37396 1 z6a] 🕸️ [status] 😅 Blacklisted [url] https://x/produkt/heavy-duty-saute-pans-50l-electric-2/
03/28/25 19:01:39.976 [103.204.128.131:37396 1 z6a] 🕸️ [status] 😅 Blacklisted [url] https://x/produkt/crepeplade/
https://hizliresim.com/orlowh8
Plugin Support
qtwrk
(@qtwrk)
please try this , go to General -> Server IP , add your origin IP , then crawl again
I tried as you said. But the problem is still not solved.
Plugin Support
qtwrk
(@qtwrk)
please share me report number again
Plugin Support
qtwrk
(@qtwrk)
eh , no , sorry , I mean , generate a new report
Would you like me to add Wp-admin information to your report?
Plugin Support
qtwrk
(@qtwrk)
no , just report should be enough
Plugin Support
qtwrk
(@qtwrk)
please try this
edit file wp-content/plugins/litespeed-cache/src/crawler.cls.php , around line 1032, you will see if ($code == 201) { insert this line before it
error_log(print_r($header, true) ,3 , '/path/to/your/wordpress/lscwp.log');
then crawl again , after that, you should see this lscwp.log file, share it via https://pastebin.ubuntu.com/
