Comments appearing 2x in single() pages.

I did some hacking to have my comments on my index pages, you can see the result here. I don’t want that slider when I’m in the post’s (single()) page, so this is how I coded theloop.php:

<?php if (!is_single()) { ?>
<b href="#" onclick="toggle_panel(this, 'comment_panel-<?php the_ID(); ?>', 'Hide comments (<?php comments_number('0', '1', '%'); ?>)', 'Show/Add Comments (<?php comments_number('0', '1', '%'); ?>)', 0.3); return false;">Show/Add Comments (<?php comments_number('0', '1', '%'); ?>)</div>
<div id="comment_panel-<?php the_ID(); ?>" style="display: none;"><div id="sliderhack-<?php the_ID(); ?>"><?php $wp_query->is_single = true; comments_template(); $wp_query->is_single = false; ?></div></div> <?php } else {comments_template(); } ?>

Works like a charm. Well, almost. If you go to an individual page, like this one, you’ll get the comments twice. Why is it doing that? How can I fix it?

Edit: changed the a tag to b in the code so it wouldn’t be interpreted by the forum.