So I set up the mathjax-latex plugin, configured to use MathJax's CDN, and everything works fine. Until, I decided to tweak a couple of MathJax's settings. And everything worked like I hoped... if I was logged into wordpress! Otherwise, I only got the math's source code, but no properly rendered formulas. After a lot of hammering, I noticed that the only way to "fix" this was to "edit" the posts that contain the math: essentially just go to the editing interface, and hit "Update", even if changing nothing. And presto, rendered math was back to non-logged-in-land.

Is this a bug (of WP)? Or, on the contrary, is it something expected? BTW, the above applies to Firefox, for instance, in chromium, I get no math, logged in or otherwise. The post with the math is here: https://blog.erroneousthoughts.org/2012/12/the-birthday-paradox/