I notice that on my test site (my own, clean theme) which has a consistent number of test comments, setting different numbers of top level comments to display affects the number of comments shown in an odd way. Sometimes only a few comments are shown on the first page (less than set in Admin), sometimes the full quota.
I can only guess that it’s something to do with the calculation of how many pages of comments to show e.g 10 top level comments divided by 3 top level comments a page, or 10 top level comments divided by 4 top level comments a page, etc.
This happens regardless of whether or not it’s last comments first, newer comments at the top etc. – I’ve tried all variations.
Changing the number of comments shown also affects whether or not previous and next page links are shown.
For example, there might be ten top level comments, and in the Admin panel it might be set to show 3 top level comments and the next / previous links will appear.
But setting the comments per page in the Admin panel to 4 might make the next / previous links disappear.