Federation speed/server bottleneck

Paul
(@happypwc)

3 days, 2 hours ago

I moved the Mastodon account @hashtaggames@oldfriends.live that I have used since 2022 for the popular 9 o’clock HashTag Games from my self-hosted Mastodon instance to a dedicated WordPress ActivityPub game domain user at @hashtaggames@hashtaggames.online with plans to better mange and archive the games. The account has around 2400 followers and follows none.

Unfortunately, having several hundred users waiting at 8:55pm EDT each day from a wide spathe of ActivityPub federated servers is bogging down my WordPress instance, slowing down federation, etc.

They are posting, commenting, interacting, fav’ing, boosting, etc the WordPress post all at the same time.

I have looked through the forums, GitHub issues, asked through the fediverse, and so far, I have made sure I have robust caching and I changed ACTIVITYPUB_OUTBOX_PROCESSING_BATCH_SIZE from the default 100 to 600 in wp-config.php and those seem to help.

I did setup wp-control so I could get a sense of what is happening.

Checking Cron Events shortly after sending out the daily post I get an cURL issue which goes away after federation is done sending the post.

cURL error 28: Operation timed out after 3000 milliseconds with 0 bytes received This means there is a network connectivity problem preventing your server from performing "loopback" requests to itself More information takes you here: Problems spawning a call to the WP-Cron system | WP Crontrol

Any help or suggestions would be greatly appreciated to stop this bottleneck.

Off the bottleneck topic, for some reason, fediverse avatars no longer appear in the posts when people interact but they do appear in the followers list in the plugin settings.

The page I need help with: [log in to see the link]

Viewing 12 replies - 1 through 12 (of 12 total)

Plugin Author Matthias Pfefferle
(@pfefferle)

2 days, 10 hours ago

Hey @happypwc 👋

I am experimenting with different distribution modes atm. https://github.com/Automattic/wordpress-activitypub/pull/3044 but with 2400 followers, the plugin should not slow down federation too much, to not need a full day to federate the post.

Does your WordPress hosting have similar capacities/performance than your former Mastodon hosting?
Thread Starter Paul
(@happypwc)

2 days, 2 hours ago
Thanks @pfefferle. To clarify, it isn’t taking a day to federate. I make one blog post per day and at the same time every day. I have a lot of people on Mastodon and other AP software in the Fedi waiting for it every day at that time from around the world, sharing, interacting, etc. It’s only taking a few minutes now, although it is taking some instances as long as 30 minutes before they get it. But, when it is federating, the server is bogged down.

The server actually has double the CPU cores and RAM. 12GB RAM. I noticed in that period, just after the post goes out, for a few minutes when it is bottlenecked, that html only pages work and things on the server that rely on PHP and MYSQL, like FreshRSS, won’t load. I am now wondering if it is either PHP or MYSQL settings. I disabled as many WordPress plugins that I don’t need at this point, not that I had a lot, and ready to troubleshoot tonight when the post goes out live when it is under a lot of live federation interaction. Even though health is good, I wonder what I am missing in any php or mysql config that would be affected by what WordPress ActivityPub plugin (or maybe Friends?) is doing (not blaming them, of course.)
- This reply was modified 2 days, 2 hours ago by Paul.
Plugin Author Matthias Pfefferle
(@pfefferle)

2 days, 2 hours ago

To clarify, it isn’t taking a day to federate

I know, but if you slow down publishing and make batches smaller, that could be the result.

Plugin Author Matthias Pfefferle
(@pfefferle)

2 days, 1 hour ago

The server actually has double the CPU cores and RAM. 12GB RAM. I noticed in that period, just after the post goes out, for a few minutes when it is bottlenecked, that html only pages work and things on the server that rely on PHP and MYSQL, like FreshRSS, won’t load.

It is mainly the traffic. All servers that gets pinged try to validate the ping and verify the signatures, so the bottleneck could also be the webserver setup and a possible limitation of parallel requests and the then called PHP processes.

Plugin Author Matthias Pfefferle
(@pfefferle)

2 days, 1 hour ago

Do you use frontend caching? Maybe that could also help!? Be sure to choose a compatible plugin or to disable Content-Negotiation in the Advanced Settings if the caching does not support it.

https://github.com/Automattic/wordpress-activitypub/wiki/Caching

Thread Starter Paul
(@happypwc)

2 days, 1 hour ago

Do you use frontend caching?

It doesn’t look like the one I was using is on the list so I will try WP-Optimize. What is the best lifespan setting for caching, noting I am using it primarily for federation activity? Also, is there a preferred or best cache plugin for AP?

Plugin Author Matthias Pfefferle
(@pfefferle)

1 day, 10 hours ago

I have good experiences with Surge so far. I will work on a more detailed guide to use caching in combination with ActivityPub, to go a bit more into the details with Content Negotiation and how to properly setup the ActivityPub plugin to not ran into caching issues. I hope I can make it today or tomorrow.

Plugin Author Matthias Pfefferle
(@pfefferle)

1 day, 8 hours ago

Let me know if it helps and if you miss something or if something does not work as expected: https://github.com/Automattic/wordpress-activitypub/blob/trunk/docs/how-to/caching.md

Thread Starter Paul
(@happypwc)

1 day, 3 hours ago

@pfefferle, I went ahead and installed Surge, and thanks for the caching.md. I’ll post back after about how it is going.

Thread Starter Paul
(@happypwc)

1 day, 2 hours ago

@pfefferle Site health reports 1 critical issue, “Surge isn’t currently set up to work with ActivityPub. While this isn’t a major problem, it’s a good idea to enable support. Without it, some technical files (like JSON) might accidentally show up in your website’s cache and be visible to visitors.” define( 'WP_CACHE_CONFIG', '/home/p37307/hashtaggames.online/wp-content/plugins/activitypub/integration/surge-cache-config.php' );

Caching.md states, Surge, Best option. The ActivityPub plugin automatically configures Surge to cache HTML and JSON as separate variants. This is the only plugin that properly caches both response types. wordpress-activitypub/docs/how-to/caching.md at trunk · Automattic/wordpress-activitypub

Plugin Author Matthias Pfefferle
(@pfefferle)

1 day, 2 hours ago

The plugin should do it automatically, maybe you have to deactivate and reactivate the ActivityPub plugin one time. I will check why the AP plugin is not triggered by the installation of surge. Maybe that is not possible through hooks!?

Thread Starter Paul
(@happypwc)

1 day ago

maybe you have to deactivate and reactivate the ActivityPub

That worked

Viewing 12 replies - 1 through 12 (of 12 total)

You must be logged in to reply to this topic.

Tags