Pagination within category's posts

I’ve 4 categories and each of it has between 10 and 40 posts.
I activated the prev-next pagination on posts, but when I arrive at the last/first post of category X, the pagination doesn’t loop within the category, but it shows the first/last of the prev/next category.

How can I make the pagination looping inside the category the current post is?
Example: I’m on a post in category ‘Name 1’, and I’d like to have the pagination that goes between the next and previous post of the category ‘Name 1’ again.

This is how my code looks like – but still, it won’t go within the category.

<?php next_post_link('<div class="single-nav-left col span_12 clr-margin">%link</div>', '&larr; %title', TRUE); ?>
<?php previous_post_link('<div class="single-nav-right col span_12">%link</div>', '%title &rarr;', TRUE); ?>