In addition to the information ergate has provided, there also seems to be pertinent code in wp-admin/admin-functions.php
The code that ergate referenced in wp-admin/inline-uploading.php appears to determine the size of the thumbnail file that WP generates, but from what I can tell, does not actually affect the dimension attributes WP uses when inserting the image. For example, changing the code in inline-uploading.php will result in a thumbnail file with larger dimensions on your server, say 256x192, but when browsing and inserting the thumbnail into the post, WP will still apply the code using something like <img src="pic.jpg" width="128" height="96 />. This is where admin-functions.php comes into play.
Lines 1771-1778 look something like this:
function wp_shrink_dimensions($width, $height, $wmax = 128, $hmax = 96) {
if ( $height <= $hmax && $width <= $wmax )
return array($width, $height);
elseif ( $width / $height > $wmax / $hmax )
return array($wmax, (int) ($height / $width * $wmax));
else
return array((int) ($width / $height * $hmax), $hmax);
}
and lines 1863-1870 look something like this:
function get_udims($width, $height) {
if ( $height <= 96 && $width <= 128 )
return array($width, $height);
elseif ( $width / $height > 4 / 3 )
return array(128, (int) ($height / $width * 128));
else
return array((int) ($width / $height * 96), 96);
}
Now I've tinkered around with it for a while, replacing the 128 and 96 values with larger integers. I have had limited success in which inserted images now appear with their original dimensions; however, when opting to insert thumbnails, the "thumbnails" are also stretched to the larger dimensions of the original, which causes them to be massively pixelated.
I am by no means up to snuff on PHP and coding in general, so I have a feeling that if someone more knowledgable were to take a look at the source in these files, there is a way to alter the dimension integers appropriately so that original images and thumbnails are displayed with their correct dimensions respectively.
Hopefully we can get this issue resolved soon.
I have also contributed to an already-established ticket on Trac that documents this same problem: http://trac.wordpress.org/ticket/2199