I did a little research on this and the wp_list_authors() tag and I can't find a way to do this, though it might be possible with the use of some conditional tags. You can read about those in the WordPress Codex at http://codex.wordpress.org/Conditional_Tags to see if there is something there you can use. There are some conditional author tags, but I don't see how they will work in this case, but who knows.
Honestly, unless you really hate them, if you want their work to still be accessed, does it really matter that it shows up? You can also change their author names to a common name like "Others" and hard code their names in the article itself so it doesn't appear on the author list.