Litespeed crawler is not working at all

Resolved tusharrao
(@tusharrao)

3 years ago

Hi,

Report: TQZVUJHG

The litespeed crawler is not working at all I am trying to crawl it i have tried troubleshooting many ways but it just doesn’t work. We’re using cyberpanel + Litespeed and we expected everything to work hand in hand.

Because of this the visitors cannot see new refreshed page. However the page caches only when the user actually visits the page and this page once stored never refreshes.

Viewing 15 replies - 1 through 15 (of 15 total)

Plugin Support qtwrk
(@qtwrk)

3 years ago

Hi,

please screenshot me your crawler status page

Best regards,

Thread Starter tusharrao
(@tusharrao)

3 years ago

Hi,

https://prnt.sc/10yibls so far nothing is hitting cache.

Regards

Plugin Support qtwrk
(@qtwrk)

3 years ago

I don’t think it is running , now please go to [2] map , do you see any URL there ? and in [3] blacklist , do you see any URL there ?

Thread Starter tusharrao
(@tusharrao)

3 years ago

No nothing there no URL in Map or Blacklist.

Plugin Support qtwrk
(@qtwrk)

3 years ago

go to Map , click “refresh”

Thread Starter tusharrao
(@tusharrao)

3 years ago

Refreshed but nothing also i can see notice “No valid sitemap parsed for crawler.” but the site map is present also i tried mentioning the link but doesnt work.

Plugin Support qtwrk
(@qtwrk)

3 years ago

please follow this guide to grab the log when it tries to refresh the sitemap

it should log more detail information why/what happened

Thread Starter tusharrao
(@tusharrao)

3 years ago

Hi,

Below is the log.

03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] 💓 ——GET HTTP/1.1 (HTTPS) /wp-admin/admin.php
03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] Query String: page=litespeed-crawler&LSCWP_CTRL=crawler&LSCWP_NONCE=8a15bb5616&litespeed_type=refresh_map
03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] HTTP_REFERER: https://xxxxx.com/wp-admin/admin.php?page=litespeed-crawler
03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99;role_exclude_cache:1
03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] LSCACHE_VARY_COOKIE: wp-postpass_0c4a259829633f08e9a633a47513ae93
03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] LSCACHE_VARY_VALUE: +webp
03/29/21 04:37:51.814 [149.202.98.186:18352 1 iGf] [Router] LSCWP_CTRL: crawler
03/29/21 04:37:51.814 [149.202.98.186:18352 1 iGf] [Router] LSCWP_CTRL verified: ‘crawler’
03/29/21 04:37:51.815 [149.202.98.186:18352 1 iGf] 🐞 Init
03/29/21 04:37:51.815 [149.202.98.186:18352 1 iGf] [Router] parsed type: refresh_map
03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] 🐞🗺️ failed to read sitemap: cURL error 6: Could not resolve host: xxxxx.com
03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] 🐞🗺️ ❌ failed to parse custom sitemap: Failed to remote read https://xxxxx.com/sitemap_index.xml
03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] 🐞🗺️ Truncate sitemap
03/29/21 04:37:51.851 [149.202.98.186:18352 1 iGf] 🐞🗺️ Generate sitemap
03/29/21 04:37:51.868 [149.202.98.186:18352 1 iGf] [Ctrl] not cacheable before ctrl finalize
03/29/21 04:37:51.868 [149.202.98.186:18352 1 iGf] [Router] get_role: administrator
03/29/21 04:37:51.869 [149.202.98.186:18352 1 iGf] 💰 X-LiteSpeed-Cache-Control: no-cache
03/29/21 04:37:51.869 [149.202.98.186:18352 1 iGf] [Optm] bypass: Not frontend HTML type
03/29/21 04:37:51.869 [149.202.98.186:18352 1 iGf] End response
——————————————————————————–

03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] 💓 ——GET HTTP/1.1 (HTTPS) /wp-admin/admin.php
03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] Query String: page=litespeed-crawler
03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] HTTP_REFERER: https://xxxxx.com/wp-admin/admin.php?page=litespeed-crawler
03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99;role_exclude_cache:1
03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] LSCACHE_VARY_COOKIE: wp-postpass_0c4a259829633f08e9a633a47513ae93
03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] LSCACHE_VARY_VALUE: +webp
03/29/21 04:37:52.648 [149.202.98.186:19084 1 v0K] [Ctrl] X Cache_control -> no Cache ( Admin page )
03/29/21 04:37:52.744 [149.202.98.186:19084 1 v0K] 🐞 Init
03/29/21 04:37:52.766 [149.202.98.186:19084 1 v0K] [Ctrl] not cacheable before ctrl finalize
03/29/21 04:37:52.766 [149.202.98.186:19084 1 v0K] [Router] get_role: administrator
03/29/21 04:37:52.766 [149.202.98.186:19084 1 v0K] 💰 X-LiteSpeed-Cache-Control: no-cache
03/29/21 04:37:52.767 [149.202.98.186:19084 1 v0K] [Optm] bypass: Not frontend HTML type
03/29/21 04:37:52.767 [149.202.98.186:19084 1 v0K] End response
——————————————————————————–

03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] 💓 ——GET HTTP/1.1 (HTTPS) /wp-admin/admin.php
03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] Query String: page=litespeed-toolbox
03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] HTTP_REFERER: https://xxxxx.com/wp-admin/admin.php?page=litespeed-toolbox
03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99;role_exclude_cache:1
03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] LSCACHE_VARY_COOKIE: wp-postpass_0c4a259829633f08e9a633a47513ae93
03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] LSCACHE_VARY_VALUE: +webp
03/29/21 04:37:58.878 [149.202.98.186:24752 1 cML] [Ctrl] X Cache_control -> no Cache ( Admin page )

Plugin Support qtwrk
(@qtwrk)

3 years ago

Hi,

03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] 🐞🗺️ failed to read sitemap: cURL error 6: Could not resolve host: xxxxx.com

this was the issue

https://sw_s_.com/sitemap_index.xml

was your sitemap setting ?

try curl this URL fro your server SSH first , see if it works there

Thread Starter tusharrao
(@tusharrao)

3 years ago

This is from local but server i have not yet tried.

<?xml version=”1.0″ encoding=”UTF-8″?><?xml-stylesheet type=”text/xsl” href=”//xxxxx.com/wp-content/plugins/wordpress-seo/css/main-sitemap.xsl”?>
<sitemapindex xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″>
<sitemap>
<loc>https://xxxxx.com/post-sitemap.xml</loc>
<lastmod>2017-06-16T08:04:22+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/page-sitemap.xml</loc>
<lastmod>2021-03-28T09:44:28+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/product-sitemap.xml</loc>
<lastmod>2021-03-16T23:14:18+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/category-sitemap.xml</loc>
<lastmod>2017-06-16T08:04:22+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/xxxxxxxxx_slider-sitemap.xml</loc>
<lastmod>2019-01-10T16:29:35+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/product_cat-sitemap.xml</loc>
<lastmod>2021-03-16T23:14:18+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/product_tag-sitemap.xml</loc>
<lastmod>2020-07-28T10:45:05+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://xxxxx.com/pa_pt_phone_model-sitemap.xml</loc>
<lastmod>2020-06-06T13:26:16+00:00</lastmod>
</sitemap>
</sitemapindex>

Plugin Support qtwrk
(@qtwrk)

3 years ago

yes, I know , it loads by external

maybe your server has DNS resolver issue or something , that’s why I asked you to curl it on server first

zorbs
(@zorbs)

3 years ago

Hello – I have the exact same issue. XML sitemap curls fine on the server.

zorbs
(@zorbs)

3 years ago

Log indicates “…failed to parse custom sitemap: Failed to parse xml {URL}”

Plugin Support qtwrk
(@qtwrk)

3 years ago

Hi,

@zorbs please create your own topic for your issue with debug log as above process

Best regards,

Plugin Support qtwrk
(@qtwrk)

3 years ago

Hi,

I’m going to mark this topic “Resolved”, due to lack of activity.

If you still need help, please feel free to re-open it.

When re-open it, please also change the topic status to “not solved”

Best regards,

Viewing 15 replies - 1 through 15 (of 15 total)

The topic ‘Litespeed crawler is not working at all’ is closed to new replies.

Tags