I recently migrated my forums from vBulletin to bbPress, and after purging all users with zero posts, I still have over 17,000 users. This wreaks havoc with several areas in WordPress that are not coded to handle large numbers of WordPress user accounts.
Particularly painful is the Widget Visibility. If I configure any widget visibility on wp-admin/widgets.php, this Jetpack module virtually renders that admin page unusable. For some reason it loads a list of ALL users, so if I view source on the page, there are over 23,000 lines of HTML.