I can give you a little more information, but I'm not sure how helpful it will be. Google's viewer does have a size limitation of its own. I am not sure precisely what this is (I've never found it published, only approximately by trial and error), but certainly if the file is too large to be displayed in the Standard Viewer, the Enhanced Viewer will not do anything to improve upon that.
If it does load in Standard but not in Enhanced, that should indeed fall under the purview of the settings you tried to change. The Enhanced Viewer basically buffers a copy of the Standard Viewer locally and makes some modifications to it before outputting it to the browser. I would not think that how the file itself loads would be affected by this, so I would expect the same behavior from both as far as file size limitation, but perhaps there's another variable there I'm not considering.
Using <embed> would do away with this, as it's not actually loading the document and doing any conversion to images (which is what the viewer does), but simply inserting a frame with the file as it is. But as you mentioned all the benefits of using either viewer are lost in this case.
Probably least helpful of all, but permit me to ask: why are your PDF files so large? If it's possible for you to optimize them for web/screen use, even if you must link to a full res version for some other reason such as "download hi-res" or "print-ready version", that would seem to make sense. I'm not suggesting you do things my way in order to "fix" the plugin, but if you haven't critically considered optimizing your PDF for web use prior to loading it into the plugin, I think you'll find a lot better results that way.
I hope that helps at least somewhat.