I'm trying to think of what it can be. Right now I don't know. I guess the above tests told us your mp3 file isn't the problem, your server isn't the problem if you tried serving the mp3 from the domain I gave above, and your computer/connection isn't the problem.
This isn't just super fun, but I guess you could try installing another plugin, this one,
http://wpaudioplayer.com/
It is pretty much the same audio player that is found in PodPress, but it doesn't have all the other features PodPress has, and it has different instructions for telling the player which audio file (doesn't have the drop-down menu thing).
I'm not saying switch to it (I prefer PodPress), but maybe install it and throw an mp3 in it, and see if you still get the same buffer problem. If you do, then the problem is not in PodPress per-se, but possibly in the Flash player or somthing else. If you don't get the problem then, you'd at least have a way to put mp3 on the site for now, and you would suspect that the problem is in PodPress code.