I just found this and am also annoyed. The thing I'd point out is that, for floated elements, it's a requirement of the CSS2 spec that an explicit width be specified, so WordPress is doing the right thing by specifying a width. However, hardcoding the extra 10px being added on deep in the application is totally braindead!
I would definitely mark this as a bug until it's removed. Here's a patch to fix it:
--- media-old.php 2008-11-21 18:01:11.000000000 -0800
+++ media.php 2008-11-21 18:01:38.000000000 -0800
@@ -372,7 +372,7 @@
if ( $id ) $id = 'id="' . $id . '" ';
- return '<div ' . $id . 'class="wp-caption ' . $align . '" style="width: ' . (10 + (int) $width) . 'px">'
+ return '<div ' . $id . 'class="wp-caption ' . $align . '" style="width: ' . (int) $width . 'px">'
. $content . '<p class="wp-caption-text">' . $caption . '</p></div>';
I wrote a little bit more about the CSS requirement on this bug report: