Accidental maximum stress test

  • Resolved solarbird

    (@solarbird)


    3 months ago

    Through a combination of bad luck and spam reduction measures, I have created a possibly unique stress test and mostly want to know if I’m going to be able to dig my way out of this.

    This current weekend, I’d planned to go through a very, very large blog and toggle off linkbacks/pingbacks because they’re entirely spam vectors.

    I set up a task to mostly automate this, and ran it…

    …just as a post I’d made about HHS workers went viral. (Around 180,000 impressions based on maths.)

    I have learned as part of this that ActivityPub considers turning off linkbacks/pingbacks to be notification-triggering edits. This has created a situation, wherein ActivityPub is now at least a day and a half behind. Replies aren’t federating, new posts aren’t federating, but occasionally, another edit notification does federate, so I think it’s just working its way – intensely slowly – through a task list which is obscenely long.

    However, at the rate it’s happening, I have no idea how many days or weeks – or months – it will be before it catches up.

    My request: …can I do anything to fix this? To make this go faster? Anything?

    My suggestion: disabling trackback/pingback should not generate an ActivityPub event.

    The page I need help with: [log in to see the link]

Viewing 15 replies - 1 through 15 (of 89 total)
1 2 3 4 5 6
  • Plugin Author Matthias Pfefferle

    (@pfefferle)

    3 months ago

    Hey @solarbird 👋

    oh no! 🫣

    you should do two things:

    1. clear the scheduler list. you can do that for example with this plugin: https://wordpress.org/plugins/wp-crontrol/
    2. You can also clean your db and remove all posts with the post type ap_outbox .

    My suggestion: disabling trackback/pingback should not generate an ActivityPub event.

    The problem with that suggestion (and we already had the same issue with Webmentions) is, that there are quite a lot of users, that want to disable pingbacks/trackbacks completely because of spam, so I would not want to tie these two features together.

    I will see if we can implement a maintenance feature, where you can disable federation completely without deactivating the plugin. Another approach would be to simply deactivate the plugin during that time.

    Thread Starter solarbird

    (@solarbird)

    3 months ago

    Thank you! I’ll install the plugin and see what I can do with it.

    But I don’t know how to clean the database – is there UI for that or is this going into the database directly and purging entries with SQL? Because that’s terrifying. xD

    Finally, I’m confused – I don’t see how disabling pingback/trackback on posts and not sending an ActivityPub notification interact. I mean, disabling pingbacks/trackbacks is what I did. I just didn’t want edit notices for every post so affected to go out as a result.

    Thanks for replying so quickly! I will install the plugin and try to figure it out. ^_^

    Plugin Author Matthias Pfefferle

    (@pfefferle)

    3 months ago

    Finally, I’m confused – I don’t see how disabling pingback/trackback on posts and not sending an ActivityPub notification interact. I mean, disabling pingbacks/trackbacks is what I did. I just didn’t want edit notices for every post so affected to go out as a result.

    But wasn’t your request, that ActivityPub should not send anything when pingbacks are disabled?

    But I don’t know how to clean the database – is there UI for that or is this going into the database directly and purging entries with SQL? Because that’s terrifying. xD

    Would the WP-CLI be an option? I can work on a CLI call to clean up the outbox!

    Thread Starter solarbird

    (@solarbird)

    3 months ago

    No, I don’t want that!

    I just want the action of disabling pingbacks on a post – the act of turning pingbacks off – to not trigger a post edit notification.

    Right now if you turn off pingbacks/trackbacks on a post, the post gets flagged as edited and everyone who has interacted with it across the federation gets an edit notification.

    That’s what’s clogged up my outbox.

    I am suggesting that this is unnecessary, because if you do it in bulk, literally every post you have gets flagged as edited and sends out an edit notification to every single person who ever interacted with any post.

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    I dug into WP-CLI and used the stock commands to have it chug through the massive backload of cron tasks at an accelerated rate, and it did. But it’s in a bad state now, and we’re not federating… anything, as far as I can tell. It is running new cron tasks, but new posts aren’t going out.

    New posts aren’t going out. Old posts that got out can be replied to from Mastodon, and they reach the blog, but attempts to refresh the profile fail with a Errorphant, so we look like we’re in a rather bad state of some kind.

    Don’t really know where to start on this. I mean… is it valid to deactivate and then reactivate the ActivityPub plugin? I didn’t even know that was a safe operation tho’ your comments earlier on this thread imply that it is. xD

    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    Another update: I can refresh the @solarbird@solarbird.net profile now from the Mastodon admin interface. But follows are definitely broken – if I try to follow from Mastodon I get the request-pending/cancel follow request state that cannot, naturally, be resolved. And no posts federate right now.

    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    This is now also be queued… So if the queue is stuck, it blocks everything 🫣

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    But it’s in a bad state now, and we’re not federating

    Could you add more context around this? What does a bad state mean?

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland Nothing federates outward. Replies to existing posts do seem to get picked up, but nothing new goes out. Follow requests don’t work, including existing follows. I went to a second account on my Mastodon instance which also follows @solarbird@solarbird.net and that account, shows its previous follow – a follow which has been in place since the blog went up – as “pending” suddenly. But the blog thinks both accounts are already followers.

    I think this is why nothing is going out.

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    The outbox processing does rely on WP Cron getting triggered and will run all batches of inboxes that are ready to be processed at that time. Assuming there’s a constant level of traffic to the site, it shouldn’t take more than 4-5 minutes for each post to be processed, based on the amount of your followers.

    For bulk editing, when there’s a ton of posts that need processing, it’s possible that the cron process times out when it needs to go through too many events to process. Is that’s what’s happening for you?

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @obenland No, wp-cron is running, cron wp cron jobs are being queued and run. Running them manually from wp-cli works apparently normally also.

    (Yes, all of this started when I did a bulk disabling of pingback on old posts – all 8100 of them. Unfortuntely, I discovered that this meant all posts got marked as edited, which generated ActivityPub post-edit notifications for every single post. I used wp-cli to push all the cron tasks through at higher speed, it took a solid day. But we got through them, eventually.)

    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @obenland @solarbird went through old posts and updated them, so the Outbox and Schedulers might be spammed with historic data and unless it is not fully processed, new posts won’t be federated and/or followers won’t receive an Accept on a follow request.

    Thread Starter solarbird

    (@solarbird)

    2 months, 4 weeks ago

    @pfefferle I don’t know how to look at the outbox itself, that seems very reasonable to me but I don’t know how to check it.

    Currently the cron queue has 750 events in it but they’re all dated right now or later. I don’t know whether this is normal or not, but when I’ve used wp-cli to force quick processing of them, they empty out, and then new ones queue up.

    (This is down from well over 8000, many of which took 30-70 seconds to run.)

    • This reply was modified 2 months, 4 weeks ago by solarbird.
    • This reply was modified 2 months, 4 weeks ago by solarbird.
    Plugin Author Matthias Pfefferle

    (@pfefferle)

    2 months, 4 weeks ago

    @solarbird I have enabled a view on the outbox custom post type, that will be released with the next version. This would allow you to check the Outbox and to delete the entries.

    Have you checked the scheduled Crons with the plugin /wp-admin/tools.php?page=wp-crontrol ?

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 4 weeks ago

    The outbox itself is not as interesting as the cron queue, which is where the processing happens. 750 is size-able. Are those all/mostly activitypub_process_outbox events?

Viewing 15 replies - 1 through 15 (of 89 total)
1 2 3 4 5 6
  • You must be logged in to reply to this topic.