Accidental maximum stress test

  • Resolved solarbird

    (@solarbird)


    3 months ago

    Through a combination of bad luck and spam reduction measures, I have created a possibly unique stress test and mostly want to know if I’m going to be able to dig my way out of this.

    This current weekend, I’d planned to go through a very, very large blog and toggle off linkbacks/pingbacks because they’re entirely spam vectors.

    I set up a task to mostly automate this, and ran it…

    …just as a post I’d made about HHS workers went viral. (Around 180,000 impressions based on maths.)

    I have learned as part of this that ActivityPub considers turning off linkbacks/pingbacks to be notification-triggering edits. This has created a situation, wherein ActivityPub is now at least a day and a half behind. Replies aren’t federating, new posts aren’t federating, but occasionally, another edit notification does federate, so I think it’s just working its way – intensely slowly – through a task list which is obscenely long.

    However, at the rate it’s happening, I have no idea how many days or weeks – or months – it will be before it catches up.

    My request: …can I do anything to fix this? To make this go faster? Anything?

    My suggestion: disabling trackback/pingback should not generate an ActivityPub event.

    The page I need help with: [log in to see the link]

Viewing 15 replies - 46 through 60 (of 89 total)
1 2 3 4 5 6
  • Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    Hm, that’s not good.

    wp cron event list --fields=hook,next_run_gmt,args let’s you see if the args change (and progress) or just get re-queued for some reason.

    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @solarbird It seems that my Follow request got an accept!

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    this is what that looks like:

    | hook | next_run_gmt | args |
    | activitypub_async_batch | 2025-02-17 22:04:54 | [["Activitypub\Dispatcher","send_to_followers"],4536 |
    | | | 7,600,0] |
    | activitypub_async_batch | 2025-02-17 22:04:54 | [["Activitypub\Dispatcher","send_to_followers"],4536 |
    | | | 8,600,0] |
    | activitypub_async_batch | 2025-02-17 22:05:13 | [["Activitypub\Dispatcher","send_to_followers"],4536 |
    | | | 0,600,0] |
    | activitypub_async_batch | 2025-02-17 22:05:34 | [["Activitypub\Dispatcher","send_to_followers"],4535 |
    | | | 8,600,0] |
    | cron_friends_refresh_feeds | 2025-02-17 22:09:38 | [] |
    | activitypub_update_followers | 2025-02-17 22:38:23 | [] |
    | wp_privacy_delete_old_export_files | 2025-02-17 22:48:43 | [] |
    | recovery_mode_clean_expired_keys | 2025-02-18 03:48:43 | [] |
    | wp_version_check | 2025-02-18 03:48:43 | [] |
    | wp_update_plugins | 2025-02-18 03:48:43 | [] |
    | wp_update_themes | 2025-02-18 03:48:43 | [] |
    | wp_scheduled_delete | 2025-02-18 03:49:07 | [] |
    | delete_expired_transients | 2025-02-18 03:49:07 | [] |
    | wp_scheduled_auto_draft_delete | 2025-02-18 03:49:07 | [] |
    | wp_update_user_counts | 2025-02-18 03:49:07 | [] |
    | akismet_scheduled_delete | 2025-02-18 09:12:55 | [] |
    | activitypub_cleanup_followers | 2025-02-18 18:38:23 | [] |
    | activitypub_outbox_purge | 2025-02-18 21:12:10 | [] |
    | wp_delete_temp_updater_backups | 2025-02-19 20:27:46 | [] |
    | wp_site_health_scheduled_check | 2025-02-23 03:48:43 | [] |
    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland okay looks like they’re just getting re-queued, they’re cycling around in a loop.

    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @solarbird I just received the 

    Are we federating again?

    Let’s find out!

    post on Mastodon.social!

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    Okay suddenly the “Are we federating again?” post appeared. I don’t know why, I didn’t change anything.

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    WAIT sorry read the wrong number. I don’t know that for sure.

    (was : One of the events in the outbox cleared, too. )

    I replied to the post from within WordPress. So far, no good. 7 items in outbox.

    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @solarbird nice! it seems that the queue is nearly up to date now! maybe still a bit slowed down by some long running old queues, but I think it should normalize in the next view mins/hours!

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @pfefferle I… presume you’re talking about some other, internal queues we haven’t specifically examined?

    (I replied to the wordpress post that finally got out from within WordPress. It has not yet federated.)

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    | hook | next_run_gmt | args |
    | activitypub_async_batch | 2025-02-17 22:04:54 | [["Activitypub\Dispatcher","send_to_followers"],4536 || | | 7,600,0] || activitypub_async_batch | 2025-02-17 22:04:54 | [["Activitypub\Dispatcher","send_to_followers"],4536 || | | 8,600,0] || activitypub_async_batch | 2025-02-17 22:05:13 | [["Activitypub\Dispatcher","send_to_followers"],4536 || | | 0,600,0] || activitypub_async_batch | 2025-02-17 22:05:34 | [["Activitypub\Dispatcher","send_to_followers"],4535 || | | 8,600,0] |

    Hm, I’m not quite sure what to make of this, honestly. The numbers left of the 600 are unexpected. The args should be like this:
    [["Activitypub\Dispatcher","send_to_followers"],4535,600,0]

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland that’s just bad terminal word-wrap, sorry, didn’t notice, here’s current and it looks like what you expect:

    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45378,100,0] |
    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45379,100,0] |
    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45358,600,0] |
    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45367,600,0] |
    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45368,600,0] |
    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45372,100,0] |
    | activitypub_async_batch | 2025-02-17 22:32:37 | [["Activitypub\Dispatcher","send_to_followers"],45373,100,0] |
    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    Also @pfefferle I ask because a reply to that post has still not federated, an edit to the test post has not federated, and the outbox is still sitting on 7 and not changing, even though the system is idle. If it’s working through something, I’m not seeing any signs of it.

    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @solarbird there seem to be still some old ones running, that delay the queue (all batches that still use 100 as offset).

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    Here’s what’s sitting in the outbox:

    +-------+---------------------------------------------------------------------+-----------+---------------------+-------------+
    | ID | post_title | post_name | post_date | post_status |
    +-------+---------------------------------------------------------------------+-----------+---------------------+-------------+
    | 45379 | [Announce] Update: Are we federating again?Let’s find… | | 2025-02-17 14:23:44 | pending |
    | 45378 | [Update] Note: Are we federating again?Let’s find… | | 2025-02-17 14:23:44 | pending |
    | 45373 | [Announce] Create: After 29 minutes (ish), it… | | 2025-02-17 14:12:00 | pending |
    | 45372 | [Create] Note: After 29 minutes (ish), it… | | 2025-02-17 14:12:00 | pending |
    | 45360 | [Create] Note: Are we federating again?Let’s find… | | 2025-02-17 13:39:43 | pending |
    | 45358 | [Announce] Update: are we caught up? yes,… | | 2025-02-17 13:38:01 | pending |
    | 45357 | [Update] Note: are we caught up? yes,… | | 2025-02-17 13:38:01 | pending |
    +-------+---------------------------------------------------------------------+-----------+---------------------+-------------+
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    The ones with 100 are interesting, they should all be 600, if you’ve not changed that constant.

Viewing 15 replies - 46 through 60 (of 89 total)
1 2 3 4 5 6
  • You must be logged in to reply to this topic.