Team Staatic

@jkburges, after further testing, we were unfortunately not able to reproduce the issue on various testing environments. Would it be possible to provide us with temporary access to your WordPress instance (or preferably a simple testing/staging environment that shows the same issue)? If so, can you provide us with access instructions sent to contact@staatic.com? This would be extremely helpful in resolving the issue in a timely manner.

Thanks for providing us with the logs. Looking at these, it appears that you have configured six additional paths. This results in the following number of enqueued URLs each:

1. 1 URL
2. 1 URL
3. 1 URL
4. 423 URLs
5. 255 URLs
6. 0 URLs

At which position is the /wp-content/uploads path? And how is the path defined exactly?

Yes, wordpress is running in a docker container with a mounted volume from the host. The uploads directory on the host is a AWS EFS volume. Do you think this could affect the crawler? From inside the container, it looks just like any other directory AFAICT.

We’re not completely sure if this is relevant, but we’ll need to do some further testing to find out.

Ok, I will give this a try.

Perfect, thanks.

Hello @jkburges,

We’re sorry to hear that you are running into an issue causing some assets to be missing from the static version of your site.

Under normal circumstances, Staatic should detect the different versions automatically. The crawler does recognize the srcset attribute (including alternative srcset attributes like “data-srcset” and “data-wpfc-original-srcset” used by some optimization/caching plugins).

In cases where the assets are not automatically detected, having the uploads directory set as an additional path in Build Settings (with at least the “Save” checkbox checked) should normally also do the trick.

When deploying your site to S3, the generated static site is fully synchronized with the configured S3 target. This includes deleting files from S3 which are not (or no longer) part of the generated static site. Since for some reason the alternative version assets appear to be missing from the build, these are marked as “stale files” and deleted from S3 during the synchronization process.

In order to find out what is causing these assets to be missing from the build we need to have a look at the publication details.

1. Can you check whether the missing assets are listed on the Resources overview by going to the publication details and then clicking the Resources tab?
2. When publishing with “Extensive Logging” enabled from Staatic > Settings > Advanced > Logging, can you see a reference to any of the missing assets in the publication logs, by going to the publication details and then clicking the Logs tab?

Looking forward to your feedback.

Thanks for the additional information and we’re sorry to hear that the development version did not resolve the issue for you.

The “Unable to enqueue crawl url” error indicates that the crawler initiation task failed because of an earlier task failure, either due to an unknown error or timeout. This causes subsequent retries to fail immediately as a result of the previous midway failure. In this case, we expect that it is caused by the initiation task failing to complete within one of the following time limits:

1) the time limit of 120 seconds, set by Staatic using set_time_limit (most common);
2) any limit enforced by your PHP configuration (e.g. PHP max_execution_time and/or PHP-FPM request_terminate_timeout);
3) any limit enforced by the web server configuration (Apache/Nginx/Caddy/etc., request timeout).

The crawler initiation task is primarily responsible for enqueuing site URLs, which are to be processed by the crawler component during subsequent tasks. If you have a large uploads directory and configured “Additional paths” to include this directory in the build, it can take quite some time to scan everything. Especially when the directory is on an external (network) file system, as appears to be the case on your production environment. On your testing environment, scanning the uploads directory is probably much faster due to local storage, causing the task to complete without timing out.

If you would have used the wp staatic publish WP-CLI command, it would probably have worked without any issues, since this is (in most cases) not limited by these constraints.

We did make further adjustments to the background processing, to improve the reliability of publications triggered from WP-Admin. Also the default time limit has been increased to 300 seconds (5 minutes), and has been made configurable using the “Publication Task Timeout” setting under Staatic > Settings > Advanced. These changes have been included in a new beta version (1.4.2-beta2), which is now available from the WordPress plugin directory as the current Development Version.

Please go ahead and see if this version resolves the issue. If it doesn’t, can you please provide us with the Staatic diagnostics JSON file? This file can be generated by going to /wp-admin/admin.php?staatic=diagnostics and may be sent to contact@staatic.com. It contains some environment information that helps us further troubleshoot the issue (feel free to open it in a text editor to verify).

Looking forward to your feedback.

@jkburges, first of all, thanks for your interest in Staatic to improve the speed and security of your WordPress site.

Staatic uses the WP Background Processing by Delicious Brains to allow the publication process to run in the background. The crawler component processes a specific number of pages/assets per batch, to prevent exceeding memory or time limits set by PHP or the host. Depending on the ability of Staatic to increase the PHP time limit, it will use different batch sizes.

It is very possible that the live traffic, combined with the selected batch size, (slightly) exceeds the time limit on your production environment. If this is the case, the publication sub-process may end and wait for WP-Cron to restart the process. In case WP-Cron is disabled, it is not able to resume automatically.

Note that Staatic will only try to increase the PHP time limit with 120 seconds, to prevent time-outs from occurring in other places of the chain, like the web server or load balancer.

What you can try is to create a simple mu-plugin file to lower the batch sizes set by Staatic, e.g. “/wp-content/mu-plugins/staatic_batch_sizes.php”, with the following content:

<?php

add_filter( 'staatic_crawl_batch_size', function ( $batchSize ) {

return 12;

});

add_filter( 'staatic_deploy_batch_size', function ( $batchSize ) {

return 24;

});

This decreases performance a bit due to some additional overhead, but may resolve the issue on your production environment. Alternatively, enabling WP-Cron should also resolve the stall.

Hopefully this answers your questions. If not, or if you have any other questions, feel free to get back to us.