There IS a way:
Download the JW Player package from http://www.longtailvideo.com/players/jw-flv-player/ and unzip it. You only will need the player.swf file. Upload it to your site (with FTP or via the Media Upload in WordPress does not matter, as long as you remember the exact URL to the file) and use that to play the FLV file inside FancyBox like this:
<a href="http://url.to/player.swf?file=http://url.to/mymovie.flv" class="fancybox-swf">link text</a>
This way, the flash player will always be used to play the FLV file. In other words, the HTML5 capabilities of JW Player are not used. For that to happen, either the JW Player will have to be completely integrated in FancyBox or you will have to create a stand-alone page that uses the full JW Player to play your FLV file and then open that page in a FancyBox iframe like this:
<a href="http://url.to/my_flv_in_jwplayer.html" class="fancybox-iframe">link text</a>
Let me know what works out the best :)