Removing users from db who don’t have posts or comments

I need to create a query that will remove users from our db who don’t have any posts or comments. The short of it is that we migrated a custom built blog to WordPress a few months ago, but the custom built blog had users who never actually posted anything. I’m finding now, that because of the large # of users in the database (over 15K, of which less than 500 have actually ever blogged/commented), WordPress is running out of memory a lot, and it’s proving a headache.

I’ve got the current query to identify the users who haven’t posted:

SELECT wp_users.id, wp_users.user_login FROM wp_users
LEFT JOIN wp_posts ON wp_users.id=wp_posts.post_author
WHERE post_author is NULL

.. but I need to include a check to see if they’ve commented, and if I could do the delete in the same query that would be great.

Any ideas?