Crawling time, longer and longer

  • Resolved Pixelwunder

    (@pixelwunder)


    2 years, 9 months ago

    The crawling takes longer and longer. So far the crawling took about 4 hours for 65,000 pages. Now it is maybe 10 to 12 hours.

    It is also strange that I entered Run Duration 900, but it shows Last complete run time for all crawlers: 433 seconds. This value remains the same.

    Also, I noticed that wp_litespeed_crawler runs with MyISAM and even LS Cache suggests a conversion to InnoDB.

Viewing 9 replies - 1 through 9 (of 9 total)
  • Plugin Support qtwrk

    (@qtwrk)

    2 years, 9 months ago

    I don’t think DB engine will make such difference

    please do me a screenshot on your crawler summary page and also provide the report number , you can get it in toolbox -> report -> click “send to LiteSpeed”

    Thread Starter Pixelwunder

    (@pixelwunder)

    2 years, 9 months ago

    QWNMACND

    Plugin Support qtwrk

    (@qtwrk)

    2 years, 9 months ago

    em , I don’t really see anything out of ordinary , is everything else all the same as before? and since when you noticed this ?

    Thread Starter Pixelwunder

    (@pixelwunder)

    2 years, 9 months ago

    The only difference is we have changed the php version from 7.4 to 8.1 .

    Plugin Support qtwrk

    (@qtwrk)

    2 years, 9 months ago

    with my very limited knowledge , I don’t think change the PHP version will make this much difference , but if possible, please try use php74 again , see how thing goes.

    Thread Starter Pixelwunder

    (@pixelwunder)

    2 years, 9 months ago

    Sorry, but what is the point of such a tip? So that php 8.1 runs without errors, various changes were made. You can not just switch back to php 7.4, then the website no longer works properly!

    Plugin Support qtwrk

    (@qtwrk)

    2 years, 9 months ago

    okay , please go to Toolbox -> debug setting -> debug log to ON , set log level to advance , log size to 30M

    then run crawler again , after that check /wp-content/crawler.log , see how long it takes for each request to process

    Thread Starter Pixelwunder

    (@pixelwunder)

    2 years, 9 months ago

    Does the complete cache have to be deleted for this?

    Plugin Support qtwrk

    (@qtwrk)

    2 years, 9 months ago

    if you have that much page , let’s try not to purge all cache first , I would assume there might be some page is purged individually by daily operations , and let’s see how long these individual pages takes to crawl.

Viewing 9 replies - 1 through 9 (of 9 total)

The topic ‘Crawling time, longer and longer’ is closed to new replies.