page URLs for custom page template only working for pages 5 and 6 (?!?)

I have Page with permalink “/latest” that pulls in recent blog posts, recent news items (a CTP) and recent events (another CTP).

I am now trying to get paging working, just for the blog posts’ list, on this Page.

The Page’s template file is using a custom WP_Query object, and passing in the page number it gets with get_query_var( 'page' ).

Typing in good, old query strings makes the paging work:

index.php?pagename=latest&page=2,
index.php?pagename=latest&page=4,
etc…

But I want the link structure to match the rest of the site.

I am already am adding several custom rewrite rules in my theme’s functions.php, so I added this one to the top:
latest/page/?([0-9]{1,})/?$' => 'index.php?pagename=latest&page=$matches[1]'

(I tested that regex with an online tester and it gives the correct results for dozens of page numbers)

Here’s the weird behavior:

/latest still works.
/latest/page/5 works by showing what is page 5 of the blog list.
/latest/page/6 also works, and so on through latest/page/15, when I run out of posts to page through.

But /latest/page/2, /latest/page/3, and /latest/page/4 all default to the generic home (index.php) page.

I would like to use this URL structure, since it matches the paged archives, and allows me to use previous_posts_link() and next_posts_link() in my template.

My rewrite rule has the highest priority of all the rules in the WP_rewrite array. What could be causing it to fail for those 3 low page numbers?