Hi Marco, the fact that the second request is due to the W3TC working correctly. It’s the “Split by: Month” that makes the index so slow. But I suppose the option “Split by: Year” is not possible for your site seeing as many posts there… Or even if it’s possible, it would just move the slowness to the yearly sitemaps.
The only things you can do at this point:
1. try deactivating the automatic priority calculation (it should make first monthly or yearly sitemap generation a bit quicker, but will probably not affect the main index)
2. use the Cache Preload option in W3TC so that a fresh but cached version of the sitemap will be ready as soon as the bots come to index your latest posts.
This last option has theoretically the most effect but I’m not sure if W3TC includes (or can include) sitemaps in the preload options.
I’m currently working on an internal caching mechanism meant to make fresh sitemap generation more speedy while still working with all caching plugins out there. But it will never be as fast as we would like it to be on large sites like yours. At least not without loosing features as priority calculation and stuff.
Please let me know if/how you can make the Preload option in W3TC to include the main sitemap index and maybe the monthly sitemaps too 🙂
Hi Marco, the fact that the second request is due to the W3TC working correctly.
Ok
It’s the “Split by: Month” that makes the index so slow. But I suppose the option “Split by: Year” is not possible for your site seeing as many posts there… Or even if it’s possible, it would just move the slowness to the yearly sitemaps.
Yes! I wrote to you in the past for this reason (https://wordpress.org/support/topic/one-sitemap-returns-white-screen-of-death). You suggested me to split by month.
The only things you can do at this point:
1. try deactivating the automatic priority calculation (it should make first monthly or yearly sitemap generation a bit quicker, but will probably not affect the main index)
Yes I think this too. But I’ll try.
2. use the Cache Preload option in W3TC so that a fresh but cached version of the sitemap will be ready as soon as the bots come to index your latest posts. This last option has theoretically the most effect but I’m not sure if W3TC includes (or can include) sitemaps in the preload options.
Unfortunally I can’t include/exclude content in cache preload settings (see https://pasteboard.co/GX69FgY.png). But I got an idea: why not schedule a call to the sitemap using the schedule tasks of the provider? I could plannify a call every 5 minutes for example. Do you think it could be a good idea?
I’m currently working on an internal caching mechanism meant to make fresh sitemap generation more speedy while still working with all caching plugins out there. But it will never be as fast as we would like it to be on large sites like yours. At least not without loosing features as priority calculation and stuff.
Thank you the same for the effort. I really appreciate it.
Thank you!
Marco
Hi Marco, what you could do is create a static sitemap.xml file manually that holds references to all the actual live sitemaps and the main index. Then upload that file to your media library and come the URL where it is stored.
Then paste that URL in the W3TC field that says “Sitemap URL” on the Cache Preload settings. From then on, W3TC should start preloading all sitemaps. Note that you will need to add each new moth manually to that static sitemap and upload it again…
So yeah, maybe a cron job calling the sitemap regularly might be easier. Still, you’ll need to add each new month manually.
The interval can be the same as the cache expiration time, as long as W3TC always serves a cached version while regenerating a new version in the background. Not sure if this is a dedicated setting or if the plugin already does this by default. I know in WP Super Cache for exampe there is an option for such behavior while Hummingbird always does this (I believe) without option to switch it off.
hi Ravan, thanks for the kind support and the good idea. We will evaluate it!
(a little update: I tried to disable the priority calculation, but the time of generation of the main sitemap is still about the same – 15.53s ; it was predictable)
