Accidental maximum stress test

  • Resolved solarbird

    (@solarbird)


    2 months, 3 weeks ago

    Through a combination of bad luck and spam reduction measures, I have created a possibly unique stress test and mostly want to know if I’m going to be able to dig my way out of this.

    This current weekend, I’d planned to go through a very, very large blog and toggle off linkbacks/pingbacks because they’re entirely spam vectors.

    I set up a task to mostly automate this, and ran it…

    …just as a post I’d made about HHS workers went viral. (Around 180,000 impressions based on maths.)

    I have learned as part of this that ActivityPub considers turning off linkbacks/pingbacks to be notification-triggering edits. This has created a situation, wherein ActivityPub is now at least a day and a half behind. Replies aren’t federating, new posts aren’t federating, but occasionally, another edit notification does federate, so I think it’s just working its way – intensely slowly – through a task list which is obscenely long.

    However, at the rate it’s happening, I have no idea how many days or weeks – or months – it will be before it catches up.

    My request: …can I do anything to fix this? To make this go faster? Anything?

    My suggestion: disabling trackback/pingback should not generate an ActivityPub event.

    The page I need help with: [log in to see the link]

Viewing 14 replies - 76 through 89 (of 89 total)
1 2 3 4 5 6
  • Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 3 weeks ago

    All cleared up now?

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    Two still in outbox, the two oldest ones.

    | 45360 | [Create] Note: Are we federating again?Let’s find… | | 2025-02-17 13:39:43 | pending |
    | 45357 | [Update] Note: are we caught up? yes,… | | 2025-02-17 13:38:01 | pending |
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 3 weeks ago

    Are the events for it processed?

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    @obenland yeah there’s no ap events in the queue right now.

    So I don’t know why there aren’t given there are two outbox items so are they waiting for an outbox reprocess sweep?

    • This reply was modified 2 months, 3 weeks ago by solarbird.
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 3 weeks ago

    Those would be good candidates for wp cron event schedule activitypub_process_outbox now --0=<post_id> then, to get re-queued.

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    @obenland Ah! Of course. Done.

    (I am going to write up so many notes after this. It’s been educational and I appreciate that)

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 3 weeks ago

    Yeah, same here, there’s a lot for us to build on.

    Thank you so much for reporting this and your patience and understanding throughout—much appreciated!

    These are some of the take aways to improve for the plugin, based on this experience:

    • Process the first batch right away, instead of queuing a new event for it, to help cut down on events to process.
    • Skip update events for comment and ping status changes, maybe others. Add a filter to extend that.
    • Skip updated events for posts that have not been federated before.
    • Find a way to check for a stampede. Maybe to check for bulk editing. @solarbird Did you bulk edit these posts through the wp-admin UI or some other way?

    Anything else I’m missing?

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    @obenland The big one is definitely the second in your list, from my POV. I really didn’t expect that to send update events – particularly for the majority of these posts which had never even been federated!

    …and that might be something else to check for. It looks to me like every post generated an update, even if it had been created before installing and enabling ActivityPub. So it wouldn’t’ve had a “do not federate” status – or any status for that field at all. Most of the posts on here are imported from other blogs including Dreamwidth and Livejournal, and I put them all in place appropriately backdated before installing AP.

    (I am, as far as I know, the only person to successfully import from a full Dreamwidth export. I put my code on github. It is… fugly. But it worked!)

    I disabled pingback/trackback through the post quick edit interface – select all items on a post list page with the number per page cranked up. Select group edit, change just the one setting, hit save. I knew it’d take some time to process, but that it would all get queued up, and that part worked as I expected. No more pingback spam. I just did not think about activitypub! xD

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    hm hm hm

    last item in outbox is an update to a post that never federated properly and it doesn’t seem to want to process. should I mark it the same way we marked the 9000?

    | 45357 | [Update] Note: are we caught up? yes,… | | 2025-02-17 13:38:01 | pending |

    WAIT NO it finally launched a crontask, let’s see what it does…

    • This reply was modified 2 months, 3 weeks ago by solarbird.
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 3 weeks ago

    Thanks! I amended my list with the not-federated-before check.

    should I mark it the same way we marked the 9000?

    Yes, that’s an option, definitely. Another thing to consider would be to bring back the reprocessing job, to pick up these stray leftover posts.

    wp cron event schedule activitypub_reprocess_outbox now hourly

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    Outbox EMPTY! 😀

    $ ./wp post list --post_type=ap_outbox --post_status=pending
    +----+------------+-----------+-----------+-------------+
    | ID | post_title | post_name | post_date | post_status |
    +----+------------+-----------+-----------+-------------+
    +----+------------+-----------+-----------+-------------+
    $
    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    @obenland Okay! And now that outbox is empty I will make it presumably not empty? by bringing back reprocess_outbox as per your suggestion. xD

    • This reply was modified 2 months, 3 weeks ago by solarbird.
    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 3 weeks ago

    By posting new posts! 🙂

    activitypub_reprocess_outbox is just a helper in case a processing event fails in unexpected ways.

    Thread Starter solarbird

    (@solarbird)

    2 months, 3 weeks ago

    @obenland Gotcha. It did not, indeed, find anything to throw into the outbox or cron queue.

    Anyway!

    To close things out: Everything seems to be running normally at this point. I have been doing a little perf/tolerance testing and ended up boosting the php run time to 60 seconds before testing to verify I could process an edit to completion with a queue length of 600.

    Upon discovering that I could, I then set the queue length back down to 300 in what I hope is an excess of safety margin. This should cut this particular sort of cron event by two-thirds and hopefully make it all more manageable should a similar unfortunate event happen again. xD

    Thank you both very much again for all the help, this has been very educational, and I’m glad you got something out of it too. Have a good evening, or whatever it is where you are!

    • This reply was modified 2 months, 3 weeks ago by solarbird.
Viewing 14 replies - 76 through 89 (of 89 total)
1 2 3 4 5 6
  • You must be logged in to reply to this topic.