Hiding comments_popup_link if no comments left?

Is it possible to hide comments_popup_link if a post has no comments left?

I know about hiding it if comments are off like this:

<?php if ( comments_open() ) : ?><p>

<?php comments_popup_link('Be the first!', '1 Comment ! ', '%  Comments  !', 'comment-link', 'Sorry, no further comments on this post'); ?>
</p>
<?php endif; ?>

The reason I ask is that when there are a lot of posts without comments then it kind of drags a blog down if they all say, ‘No comments left’ or ‘Be the first’ or whatever.

So I leave the no-comments parameter empty e.g.

<?php comments_popup_link('', '1 Comment ! ', '%  Comments  !', 'comment-link', 'Sorry, no further comments on this post'); ?>

But this still outputs html, which messes up the layout. I can’t find a conditional that will do the trick – is there any code that can do this?