Don't take it personally. I don't know you anymore than you know me, so it's proven wiser to ALWAYS start at the beginning. And I'm not intentionally misunderstanding you, though you are giving bits and pieces of the information.
I understand your problem. More than you might ever believe :/ The short answer is that there is no perfect silver bullet plugin that does exactly what you want. If that's the only answer you want, and not some suggestions I've tried that have worked, and spec on others that might, then you can stop reading and know that it's not JUST a wordPress problem (go to the NY Times and watch their troll-a-thon right after some major news, it's pretty funny).
As for plugins that might help, try Report Comments -- http://wordpress.org/extend/plugins/report-comments/ -- No, it's not perfect, but you mentioned wanting a way to report comments. This one works pretty well.
This suggestion is obvious, but: Edit your theme so that, above your comment entry field, you have a disclaimer to remind people of the obvious. I know, they never read, but education can mitigate a lot of the hurt feelings. Every moderate-to-large site with a community is going to face trolls, and people (by which I mean readers) can't be coddled. They need to be TOLD 'Yo, don't feed the damn trolls!'
Which actually is why you're having a problem, alas :(
Most WP comment moderation tools are there to fight SPAM. You have trolls. It's a slight distinction, but there it is. Spammers want to pimp crap, trolls want to piss people off. And banning trolls isn't something you can easily automate like you can spammers :/ Why? Well a troll is kind of special to each site. Some people don't care and let commenters run wild, others are really firm about being on topic, and so on and so forth.
I don't know about your site, but mine has pretty common buzzwords the trolls and anti-people like to use to rile people up. I get sexual slurs alot, with the sole purpose of inspiring ire, and they're on my comment moderation list. Every time someone posts a bad-word, the post is moderated and I review it. If it's a troll, I put their name in the blacklist and carry on. That seems to cover about 90% of my pain.
Anytime I get a questionable first comment, I toss in the email address and IP into my moderation queue. Usually I can spot the weird replies easily enough -- they reply to old posts, they're using AOL.com (seriously), they don't seem to be 100% on topic for the blog post. If I even feel REMOTELY squidgy, I moderate them.
There's a comment whitelist plugin out there, but it's old and may need some manual massaging to the code: http://wordpress.org/extend/plugins/comment-whitelist/
Not wanting to require registration is not a bad thing, but it does make moderation harder, since by the very virtue of being 'open', you open yourself up to the rampant packs of idiots. Which you know. Something like Disqus may not be a bad idea, since it has those vote-up/vote-down stuff built in. Frankly, if it gets that bad where you simply cannot keep up, then you may HAVE to switch to registered users only. The sad truth is that if you get a serious troll, even moderating 10 comments won't stop 'em. A**holes will wait. Most forums require registration to post, and people seem to be okay with that. Check with your community, and be honest. "We have trolls, and one of the easier ways to stop them would be requiring registration. Would you be opposed to that?" You might be pleasantly surprised.
Should there be a better way to moderate comments? I'm torn. I don't think there's a magic number of posts where someone will prove their not a troll. I think that as soon as the poster's no longer moderated, they'll become an utter prat. It's trying to apply a code solution to a social problem, and I don't think there's ever going to be a good one :/
On the site where I do have this same problem, I stick with moderation of the first post, and over the last *mumble* decade or so of this sort of thing, I've managed to educate my commenters to shout "TROLL IN THE DUNGEON!" and then they ignore them. The new guys pick that up pretty fast.