If you insert an image WITHOUT caption and align to the left or the right of the text, no problem.
If you insert an image WITH caption and align to the left or the right of the text, no problem either.
However if you try to add a caption LATER ON to an image you aligned and inserted in the text originally WITHOUT caption, the image is thrown to the bottom of the paragraph you tried to align it with.
So it seems that there is a problem with POST CAPTIONNING of images in WordPress 3.9