Crawler problems

  • Resolved regipl

    (@regipl)


    4 years ago

    Hi, Something is not okay with my crawler, I have a lot of pages (14;000) and he never managed to cache them all.
    When i setup multiple threads for example 3 and check crawler status i see all the time only threads: 1.
    “06 Jun 2022 20:54:31 Size: 14136 Crawler: #1 Position: 1700 Threads: 1 Status: crawling, updated position”

    Crawler is also stopping without any reason..

    Any idea what is wrong here?

    RAPORT: QNMSXFWD
    06/06/2022 19:10:55

    • This topic was modified 4 years ago by regipl.

    The page I need help with: [log in to see the link]

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Support qtwrk

    (@qtwrk)

    4 years ago

    please do me a full screenshot on crawler status page

    Thread Starter regipl

    (@regipl)

    4 years ago

    Print screen:
    https://klocki.fun/status.png

    As you can see for example now, it is not moving forward..

    Plugin Support qtwrk

    (@qtwrk)

    4 years ago

    there was still half page above this one , that’s the most important part.

    Thread Starter regipl

    (@regipl)

    4 years ago

    Sorry, one more try:
    https://klocki.fun/status1.png
    https://klocki.fun/status2.png

    Thread Starter regipl

    (@regipl)

    4 years ago

    Plus Debug Log if it helps:
    https://klocki.fun/debug.txt

    Plugin Support qtwrk

    (@qtwrk)

    4 years ago

    on your screenshot 1, you can see ended reason: stopped_reset

    this means there was a purge all happened during the crawler , so crawler reset its position and wait for the interval time passes

    try set the interval to very very short time for test, like 61 seconds

    and then make sure you have cron works/triggers properly and in time.

    Thread Starter regipl

    (@regipl)

    4 years ago

    When i restart the crawler it keeps going but only after 15 min stopping. I do not see anything that would explain this in the debug log, can you pelase check?

    https://klocki.fun/error.txt

    It stopps crawling at: 06/13/22 17:47:37.352
    but i do not see any error messages.

    Plugin Support qtwrk

    (@qtwrk)

    4 years ago

    if you are checking on /wp-content/crawler.log , did you see /wp-content/debug.purge.log as well ? if so , what’s content of it ?

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘Crawler problems’ is closed to new replies.

  • 8 replies
  • 2 participants
  • Last reply from: qtwrk
  • Last activity: 4 years ago
  • Status: resolved