If you install the development version, there will be a new option "Scale large content down to fit in the browser viewport."
This will prevent the down-sizing of your long image. But the viewer will have to scroll the main window to see the rest of the image so it will not be the same as PDFs where you scroll inside the FancyBox window... The only way to accomplish the same effect as PFDs is to show the long images inside FancyBox in iframe mode.
To do this, enable the iFrame feature under the FancyBox settings and then go and change the link on your posts/pages so they look like this:
<a href="URL_TO_LONG_IMAGE" class="nofancybox fancybox-iframe">TEXT OR THUMBNAIL IMAGE</a>
Notes: The 'nofancybox' in the class is there to prevent the auto detection of images kicking in. If you have disabled auto detection, that part of the class is not needed. But only if you install the development version...