This issue has been mentioned in 2009:
Has anyone figured out a workaround when the referrer page is paginated?
The plugin works great as long as your archive/category/tag referrer pages are not paginated. If the referrer includes cat/page/1, etc., smarter navigation does not allow browsing all single posts in category.
1 Apr 2009 at 6:24 pm
What I wanted to achieve is browsing ‘post to post’ (with your pluguin) from the first to the last post in a category, regardless of the pagination. Now you can browse ‘post to post’ until the end of the page you come from (page/1, page/2, etc). I hope this makes sense now…
2 Apr 2009 at 8:45 am
Yes, now I get what you’re saying. This is definetly something worth fixing. I’ll see what I can do.