Ok, here’s what the problem is, and I think it may indicate problems on both plugins, but I’ll let you decide what can reasonably be fixed here.
1. Page Builder runs do_action_ref_array( ‘in_widget_form’, … ) in a couple places, potentially when it isn’t necessary. For example, on the Add New Post screen, it fires approximately 15 times.
2. As the author probably knows, Widget Visibility hooks onto that action each time, and attempts to add it’s visibility controls to the “widget”

Normally, that is no problem on your average site. Widget visibility adds controls for each page, category, tag, archive, and author.
If the query for any of these is too large, that’s what bogs things down. In my case, it is the authors query, as it finds 5,000+ users and parsing that 15 times takes a little while.

It is not necessary to query all users on a site, only those who are authors, and it was relatively easy to form a blacklist of non-author roles for my site like so (line 100 of src/WidVis/Admin.php):

$authors = get_users( array( 'orderby' => 'display_name', 'role__not_in' => array( 'Subscriber', 'EDD_Subscriber' ) ) );

I’m using Easy Digital Downloads on my site, thus the extra role in the exclusions list. Wouldn’t be a bad idea to keep that so other EDD users don’t have the problem, and there might be other plugins that add custom non-author roles. If you really wanted to get fancy, you could dynamically check which roles don’t have the capabilities unique to the author (and above) roles and exclude those: https://codex.wordpress.org/Roles_and_Capabilities#Author

Thank you so much for your time!