The scanner does not work

Hi @almendron

Broken Link Checker scanner scans your site’s front-end (just like any search engine crawler would – but faster) and checks all the links it can find there.

The example link that you shared is part of the scanned page (from crawler/user perspective) even if – technically speaking – it is a separate WordPress installation (or some other platform, even a pure HTML site).

If you open your “active website” all it takes to get to that example link manually is to

– click on “BITACORA” item in menu (so it takes you to /blog/ page)
– then click on “Arte” category in sidebar
– then navigate to the page number 7 (currently, it may change) using paging numbers
– and that example link is there.

It doesn’t really matter if the link “technically” belongs to the particular setup or not, it could even be an external link to some 3rd-party unrelated site. For as long as it’s available on checked site, it’s discovered and tested.

The “old” engine worked in a different way and actually checked only “content” of the page its on – so yes, it might have been skipping a lot of links in case of such “connected” installs. But still – this isn’t a bug, it’s just how crawler works.

In this specific case some of the checked URLs were responding too slow so cloud engine marked them as “Request timed out” (as it cannot wait indefinitely). After a number of such time-outs, entire scan is halted due to time out in order not to stress the site too much – so this is a security precaution to avoid breaking the site due to intense scan.

And that scan would be pretty intense. Apparently for some of the URLs found in footer, scan identified nearly 21 million (!) occurrences across the entire site – where by site I mean what “user sees from front-end” rather than single WordPress installation.

This isn’t an easy thing to address, I’m afraid, for now, unless you decide to go with the old “Local” scan again. In future we’ll be adding additional “rate limiting” to cloud scan and other options that would make it better handle such a huge amounts of links and bigger number of time-outs but I don’t have ETA on that – it’s a work in progress currently so it might take a while until it gets updated.

Kind regards,
Adam

HI @almendron

Thank you for response!

The plugin does scan all, indeed, if you use the “Cloud” scan.

Those may be three separate setups but, as I mentioned, the new “Cloud” scan works from outside. It’s like any crawler – it scans on front-end and it follows the link.

In other words – it checks the site it was started for and finds all the links there then it follows those that “from outside” appear to the part of the same side.

Your setup, despite being three separate setups, appear like a single site. Even a human user would not think it’s three separate sites. And crawler doesn’t know that too – all three sites are “interlinked” from menus, are in the same domain and use URL structure that make them appear to be single site.

It really doesn’t matter if they are different setups or not. It’s crawler that crawls the links and checks them.

Your setup is not really a typical setup as usually even site’s in sub-folders are not so tightly integrated on front-end. If there would be no links from one site to another, crawler would not follow those links. But there are, making it all appear as a single setup.

The entire Cloud engine works from outside and that’s why it sees it this way – it really does pretty much nothing on the site’s end (aside form providing user interface under “Link Checker -> Cloud (new)” page and authenticating site with the Hub to allow scan).

It’s not a bug – it’s how it is designed to work and the reason why it scans all those three sites in this case is not the scanner but how those sites are set; it’s expected and correct way to scan.

—

For now, you can always switch to “Local (old)” scan which is the very same engine (exactly the same) that it was before 2.0 plugin and it doesn’t use Cloud scanner so it will work exactly the same way as it used to work before.

In future, there will be options added also to define exclusions for Cloud scan which will make it possible to exclude certain URLs from scan; but this is planned for future and I don’t have ETA currently.

Kind regards,
Adam

Hi @almendron,

We do understand your concern and thanks for your valuable feedback and for sharing your perspective on the plugin’s functionality. I’ll make sure to bring this further to our developer’s attention to see if there are any further improvements that could be looked at regarding this.

Will keep you posted once we get further feedback.

Kind Regards,

Nithin