This is a deliberate change in WordPress 3.4. Instead of having
caption="" as an attribute of the
[caption] shortcode, WordPress inserts the caption into the main content of the shortcode, after the image and in between
This change in 3.4 was necessary to allow for HTML in captions. But, like everything we develop, it was deliberately coded so it would be backwards compatible. Any theme using the 'img_caption_shortcode' filter to override how a caption is generated would work swimmingly, even though the format changed.
If I had to guess, the common issue for all for these themes is that they call
add_shortcode( 'caption', 'some_function_in_their_theme' ); to customize the caption, instead of using the aforementioned filter. This is wrong. Themes should not be doing this. If the theme you are using does this, please contact the theme author. If they disagree it is a problem with their theme, I'd be happy to explain it to them.
I came here after seeing a bug report, but note that after two weeks and 2.5 million downloads, this is the only thread I've seen on this issue, so thankfully it does not appear to be adversely affecting many sites. I'm sorry if it has affected you. If you do know of any other threads, please link them to me.