Accidental maximum stress test

  • Resolved solarbird

    (@solarbird)


    3 months ago

    Through a combination of bad luck and spam reduction measures, I have created a possibly unique stress test and mostly want to know if I’m going to be able to dig my way out of this.

    This current weekend, I’d planned to go through a very, very large blog and toggle off linkbacks/pingbacks because they’re entirely spam vectors.

    I set up a task to mostly automate this, and ran it…

    …just as a post I’d made about HHS workers went viral. (Around 180,000 impressions based on maths.)

    I have learned as part of this that ActivityPub considers turning off linkbacks/pingbacks to be notification-triggering edits. This has created a situation, wherein ActivityPub is now at least a day and a half behind. Replies aren’t federating, new posts aren’t federating, but occasionally, another edit notification does federate, so I think it’s just working its way – intensely slowly – through a task list which is obscenely long.

    However, at the rate it’s happening, I have no idea how many days or weeks – or months – it will be before it catches up.

    My request: …can I do anything to fix this? To make this go faster? Anything?

    My suggestion: disabling trackback/pingback should not generate an ActivityPub event.

    The page I need help with: [log in to see the link]

Viewing 15 replies - 61 through 75 (of 89 total)
1 2 3 4 5 6
  • Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @obenland I think these are the long running ones, that were already started when @solarbird changed the const.

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland Yeah I changed it back briefly, then changed it back to 600 again. This was because we seemed momentarily to be working, and then we weren’t again, and I wondered if the 600 queue size was somehow causing problems. But it wasn’t, so I went back to 600 again.

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    Something is clearly rescheduling these events, I’m not sure where/how that happens. Are there other plugins/cron things that are going on?

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    I had prepared this debug code in case you have access to your php-errors log, but I’m not sure we even need it. It does look like the batches are being processed, they are just being rescheduled again somehow?

    function debug_outbox_process( $inboxes, $json, $actor_id, $outbox_item_id ) {
    	\error_log( 'Complete. Outbox item: ' . $outbox_item_id . '. Inboxes: ' . count( $inboxes ) );
    }
    add_action( 'activitypub_outbox_processing_complete', 'debug_outbox_process', 10, 4 );
    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland Certainly nothing new. We also run the Friends plugin, but that’s not new either, we’ve been running it for months. I have these plugins active:

    ActivityPub
    Advanced Excerpt
    Askimet
    Classic Editor (which I don’t even use)
    Friends
    Include Mastodon Feed
    Independent Analytics
    JournalPress (cross-poster to Dreamwidth)
    WP-Control

    and that’s it

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland That’s what it looks like to me, too – I do have access to literally everything, so I can throw that into …

    wait I just found some shit in apache’s logs

    OKAY. Now I just need to remember how to up this execution time limit.

    (Is this a side-effect of upping the queue size to 600?)

    [Mon Feb 17 14:19:20.162754 2025] [php:notice] [pid 1972963:tid 1972963] [client 50.47.64.109:33918] Cron unschedule event error for hook: activitypub_async_batch, Error code: could_not_set, Error message: The cron event list could not be saved., Data: {"schedule":false,"args":[["Activitypub\\Dispatcher","send_to_followers"],45358,600,0]}
    [Mon Feb 17 14:42:56.736577 2025] [php:error] [pid 1975434:tid 1975434] [client 50.47.64.109:58510] PHP Fatal error: Maximum execution time of 30 seconds exceeded in /home/kahvi/public_html/blog/wp-includes/Requests/src/Transport/Curl.php on line 510
    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    I’m hitting the ends of my wits, I’m afraid.

    Could you run wp cron event unschedule activitypub_reprocess_outbox to make sure that’s not what’s rescheduling these events? (That event is a safety mechanism meant to catch unprocessed outbox items when the associated events died for any reason)

    Not sure if you’re interested in the inner workings of these batch events, but they’re designed to finish up when there no more inboxes to process and publish the Outbox item. For some reason that doesn’t seem to be happening? Is that right?

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    Yes, that could definitely be a side effect of that.

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    hm, it looks like one of the inbox notifications maybe is what’s breaking it

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland Okay so first I’m bringing that back down to 100. Will the actual processing adjust back down to that on requeue?

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    On requeue, yes

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    Okay, I’ve cleared the queue of its last 600-item event, the 100-item events should be running in a few seconds, and the 600-item event should requeue itself in… how long?

    There are currently four items in outbox:

    +-------+-----------------------------------------------------------------+-----------+---------------------+-------------+
    | ID | post_title | post_name | post_date | post_status |
    +-------+-----------------------------------------------------------------+-----------+---------------------+-------------+
    | 45373 | [Announce] Create: After 29 minutes (ish), it… | | 2025-02-17 14:12:00 | pending |
    | 45372 | [Create] Note: After 29 minutes (ish), it… | | 2025-02-17 14:12:00 | pending |
    | 45360 | [Create] Note: Are we federating again?Let’s find… | | 2025-02-17 13:39:43 | pending |
    | 45357 | [Update] Note: are we caught up? yes,… | | 2025-02-17 13:38:01 | pending |
    +-------+-----------------------------------------------------------------+-----------+---------------------+-------------+

    oh, and I forgot to note: there were “No events found for hook ‘activitypub_reprocess_outbox’.”

    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    They should all be 100 item events that re-queue themselves with updated offset numbers as they go through your follower list

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    Okay, we got the test reply through at minute 59. It was clearly one of the 600-item tasks. apache’s logs are not repeating the PHP errors, as… well, one would expect. Outbox down to 3 items.

    | ID | post_title | post_name | post_date | post_status |
    | 45372 | [Create] Note: After 29 minutes (ish), it… | | 2025-02-17 14:12:00 | pending |
    | 45360 | [Create] Note: Are we federating again?Let’s find… | | 2025-02-17 13:39:43 | pending |
    | 45357 | [Update] Note: are we caught up? yes,… | | 2025-02-17 13:38:01 | pending |
    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    Nice!

Viewing 15 replies - 61 through 75 (of 89 total)
1 2 3 4 5 6
  • You must be logged in to reply to this topic.