Is it a valid approach?

  • Resolved George

    (@giorgos93)


    7 months ago

    Hi, I use your plugin for a long time, and I’ve recently noticed, that the link checking is working extremely slow (I use local version). I analyzed the problem and found the reasons:

    1. I use basic “server load limit” parameter (10.00), even though my website’s limit is always higher (usually starting from 17-18) and more.

    2. I disabled WP Cron a long time ago, and now it’s working straight from my server (hosting). It works once an hour, so the “blc_cron_check_links” cron now starts once an hour (not every 10 minutes, like it was set by you). So this cron quickly stops because of “server load limit” parameter.

    That’s why I decided to change the logic of link checking, and I wanted to ask your opinion, if it’s a valid approach?

    1. I disabled link checking, when I’m in admin-panel. And I completely removed “server load limit” parameter.

    2. I changed the option to recheck old links evey 24 hours instead of every 72 hours.

    3. I put “blc_cron_check_links” cron at 03:00 AM of my local time, because at this time there is a lowest site traffic. The cron task triggers once a day (every 24 hours).

    4. I increased the link checking process option from 420 seconds to 4600 seconds, so the process won’t stop 7 minutes later.

    My goal is to activate all links checking once every 24 hours (at 3 am), and to make sure, that nothing will interfere the process, so it can be fully completed.

    Is it a valid approach? Or I should change something?

Viewing 15 replies - 1 through 15 (of 26 total)
1 2
  • Plugin Support Amin – WPMU DEV Support

    (@wpmudev-support2)

    7 months ago

    Hello @giorgos93

    Hope you are doing well today.

    I shared the information with our broken link checker development team, it seems your approach is correct but in general, BLC should not cause high resource usage.

    We will get back to you as soon as we have further feedback.

    Best Regards
    Amin

    Thread Starter George

    (@giorgos93)

    7 months ago

    We will get back to you as soon as we have further feedback.

    Thanks! I’ll be waiting.

    BLC should not cause high resource usage.

    For some reason, it does… I noticed a couple days ago, that my link checking doesn’t work. So I removed “server load limit” parameter, and the links started to check without stoppage, because I was in admin-panel. This caused a massive increase in CPU load.

    And btw: I’ve already tried today this approach, that I’ve mentioned in previous comment. However, the link checking wasn’t fully finished after 7 minutes (I used your basic “420 seconds” parameter). So I had to run it manually, to finish it. Please note, that beforehand I activated option to delete all plugin’s data, so the scan started from no data at all (for testing purposes). Maybe at this time it’ll be faster.

    I also decreased links check from 30 seconds to 15 seconds, and disabled link check for images.

    • This reply was modified 7 months ago by George.
    Thread Starter George

    (@giorgos93)

    7 months ago

    Unfortunately, my approach doesn’t work for some reason. Today the cron task was activated again at 03:00 AM, but it checked only 1 URL from 1611 unique URLs, and doesn’t work anymore, even though I put needed settings…

    Thread Starter George

    (@giorgos93)

    7 months ago

    And couple more questions:

    1. What does blc_cron_database_maintenance cron needed for?

    2. I disabled option to check images on my website, but your plugin still collected all the images links. Is it normal, that the plugin collected them? Now it won’t check them? Or the plugin thinks, that this images are simple links? They look like this: https://website.ru/wp-content/uploads/2019/09/imagename.jpg

    Thread Starter George

    (@giorgos93)

    7 months ago

    UPD: I changed the approach, and made “blc_cron_check_links” cron work once an hour. And I noticed, that when the links checker starts, the “server load limit” parameter jumps from 20-30 up to 100, so it immediately shuts down the process.

    What causes such a high load during links checking?

    Plugin Support Amin – WPMU DEV Support

    (@wpmudev-support2)

    7 months ago

    Hello @giorgos93

    We got further feedback from our BLC team.

    I disabled link checking, when I’m in admin-panel. And I completely removed “server load limit” parameter.

    Since you moved the process to happen on a time range with low site usage it’s fine.

    I changed the option to recheck old links evey 24 hours instead of every 72 hours.

    In general, the more frequent, the better for maintaining link quality. But it can also increase resource usage. Since this happens on low-traffic hours I don’t think it’s going to be an issue.

    I put “blc_cron_check_links” cron at 03:00 AM of my local time, because at this time there is a lowest site traffic. The cron task triggers once a day (every 24 hours).

    This is great so it won’t affect site load for users.

    I increased the link checking process option from 420 seconds to 4600 seconds, so the process won’t stop 7 minutes later.

    This gives time to complete without interruption. Sensible change to make on low-traffic hours and a decent server.

    1. What does blc_cron_database_maintenance cron needed for?

    This is for plugin data cleanup, it deletes outdated date related to BLC from database.

    I disabled option to check images on my website, but your plugin still collected all the images links. Is it normal, that the plugin collected them? Now it won’t check them? Or the plugin thinks, that this images are simple links? They look like this: https://website.ru/wp-content/uploads/2019/09/imagename.jpg

    It shouldn’t check the image links if you disabled checking images in settings, may I know how know it still checks them?

    Regarding the last issue I asked our BLC team again but maybe you have too many links or there is a conflict issue, since you mentioned it was working fine before.

    Best Regards
    Amin

    Thread Starter George

    (@giorgos93)

    7 months ago

    @wpmudev-support2 thanks for the answer!

    It shouldn’t check the image links if you disabled checking images in settings, may I know how know it still checks them?

    According to your plugin’s data, there is 1611 unique links on my website. I checked the tab with all links, and image links are presented there as well, even though I disabled the option. And when I start the “blc_cron_check_links” cron, all 1611 links are starting to be checked. So I guess, the images are continuing to be checked.

    Regarding the last issue I asked our BLC team again but maybe you have too many links or there is a conflict issue, since you mentioned it was working fine before.

    Probably a conflict ussue, because now the links are continuing to be checked, and “server load limit” parameter went back to 20-30. Now I am trying to understand, what’s happening.

    Plugin Support Williams – WPMU DEV Support

    (@wpmudevsupport3)

    6 months, 4 weeks ago

    Hi @giorgos93 ,

    I hope you’re doing well.

    According to your plugin’s data, there is 1611 unique links on my website. I checked the tab with all links, and image links are presented there as well, even though I disabled the option. And when I start the “blc_cron_check_links” cron, all 1611 links are starting to be checked. So I guess, the images are continuing to be checked.

    I’m sorry, could you please share a screenshot of this with us so we can check it further in our test lab?

    You can hide any sensitive info that the screenshot could contain and you can use a service like monosnap or Google Drive but keep in mind setting the share link to public so we can access it.

    Probably a conflict ussue, because now the links are continuing to be checked, and “server load limit” parameter went back to 20-30. Now I am trying to understand, what’s happening.

    In those cases, usually, slow logs from your Hosting could help. There are other tools like NewRelic that allow you to check what is happening while the CPU load is going high, but you may need to integrate it directly into your server.

    Also, asking for help from your Hosting Support, who usually have access to more monitoring tools help.

    I hope this helps.
    Best Regards,
    Williams Valerio

    Thread Starter George

    (@giorgos93)

    6 months, 4 weeks ago

    I’m sorry, could you please share a screenshot of this with us so we can check it further in our test lab?

    Sure: https://skrinshoter.ru/sWcJ9YkFsEU . Here you can you see some URLs of images on one of pages. They are mixed up with simple URL links.

    And it happens, even though I enabled only simple URL links and text links (which are not clickable): https://skrinshoter.ru/sWcQERfAyuz

    Plugin Support Amin – WPMU DEV Support

    (@wpmudev-support2)

    6 months, 4 weeks ago

    Hi @giorgos93

    I ran some tests in my lab site using the latest version of the Broken Link Checker plugin, but I couldn’t replicate the image issue, the image link should disappear after disabling the Check image links in plugin settings.

    Could you please try to resave the settings and force-reached links in settings -> advanced? Also, please ensure you are using the latest version of the plugin.

    I changed the approach, and made “blc_cron_check_links” cron work once an hour. And I noticed, that when the links checker starts, the “server load limit” parameter jumps from 20-30 up to 100, so it immediately shuts down the process.
    What causes such a high load during links checking?

    We are unsure how the blc_cron_check_links cron job was scheduled to run hourly, as it is currently running every 10 minutes. The threshold set in the settings, which defaults to 72 hours, is used to check the condition of the links stored in the database. The links stored in the BLC table are rechecked once this threshold expires.

    We believe that a 24-hour rechecking interval is too frequent and could negatively impact performance, potentially causing CPU spikes.

    Best Regards
    Amin

    Thread Starter George

    (@giorgos93)

    6 months, 4 weeks ago

    Hi, @wpmudev-support2

    Also, please ensure you are using the latest version of the plugin.

    I do use the latest version of your plugin.

    Could you please try to resave the settings and force-reached links in settings -> advanced?

    I’ve already tried it several times before, but it didn’t help – the plugin still collects the images. But I have a theory, why it happens: the plugin mistakes images for links, because I use “Responsive Lightbox & Gallery” plugin, and it makes my images clickable – https://skrinshoter.ru/vWeAofktpLP. You can see it here, for example: https://litfan.ru/literaturnyj-slovar/chto-takoe-fanfik-prostymi-slovami/

    We are unsure how the blc_cron_check_links cron job was scheduled to run hourly

    I set it with the help of WP Crontrol plugin. Atm I make it run once every 24 hours (at 03:00 am).

    We believe that a 24-hour rechecking interval is too frequent and could negatively impact performance, potentially causing CPU spikes.

    I recently returned to 72 hours interval, and now I am waiting for the next checking to see the impact.

    Plugin Support Nebu John – WPMU DEV Support

    (@wpmudevsupport14)

    6 months, 3 weeks ago

    Hi @giorgos93,

    I hope this message finds you well.

    But I have a theory, why it happens: the plugin mistakes images for links, because I use “Responsive Lightbox & Gallery” plugin, and it makes my images clickable

    Can you please confirm if the following plugin is being used? If not, can you please share a download URL for the plugin so that we can check this issue further?

    https://wordpress.org/plugins/responsive-lightbox/

    I set it with the help of WP Crontrol plugin. Atm I make it run once every 24 hours (at 03:00 am).

    Can you please ensure there aren’t any failed reports for the crown?

    I recently returned to 72 hours interval, and now I am waiting for the next checking to see the impact.

    Please update us here with the results so that we can help you further.

    Best Regards,
    Nebu John

    Thread Starter George

    (@giorgos93)

    6 months, 3 weeks ago

    Hi, @wpmudevsupport14

    Can you please confirm if the following plugin is being used?

    Yes, I use this plugin. Btw, maybe I need to exclude a whole path from checking (/wp-content/uploads/), if it’s possible in your plugin? All the images are stored there.

    Can you please ensure there aren’t any failed reports for the crown?

    There arent’ any failed reports – the cron starts at 03:00 am. But in 3 hours (10800 seconds) it is able to check only around 600-700 links, so the process ends without being fully finished. However, if I turn on option to check links when I’m in admin-panel, then the proccess works much faster, for some reason.

    Please update us here with the results so that we can help you further.

    It doesn’t help unfortunately – even if it works every 72 hours, the checking process is extremely slow and can’t be fully finished. Maybe I need to increase some parameters of my website? For example, “memory_limit” (I have 256M), or “max_input_time” (I have 120), or “max_execution_time” (I have 1200), or some other parameter?

    Plugin Support Amin – WPMU DEV Support

    (@wpmudev-support2)

    6 months, 3 weeks ago

    Hi @giorgos93

    You are right, the responsive lightbox could be related to the images issue because it changes images source code structure. I will ask our BLC team about best way to exclude them.

    Your max execution time is above our recommendation, but your memory limit could be increased if you have many posts/pages you can try 512M for the memory limit.

    I still believe the issue might be due to a plugin conflict, and I suggest running a full conflict test on a staging site.
    https://wpmudev.com/docs/getting-started/getting-support/#conflict-test

    Best Regards
    Amin

    Thread Starter George

    (@giorgos93)

    6 months, 3 weeks ago

    Hi, @wpmudev-support2

    You are right, the responsive lightbox could be related to the images issue because it changes images source code structure. I will ask our BLC team about best way to exclude them.

    Thanks, I’ll be waiting then! Excluding images will greatly reduce a number of links for checking, so maybe it’ll help.

Viewing 15 replies - 1 through 15 (of 26 total)
1 2

You must be logged in to reply to this topic.