Hi strangebiscuit,
It seems there is nothing you can do about the curlies bieing stripped from the menu item class 🙁
Are you linking to an image from the menu?
If not, then you are using the wrong class. Use fancybox-inline for inline content, fancybox-iframe for a complete web page, etc… And make sure the corresponding option is activated on Settings > Media.
After that, you can define a default size there on Settings > Media which might be the only option you have to control the size of FancyBox from the menu.
If you ARE linking to an image, you could still use the fancybox-iframe class and use its default size to control the FancyBox frame 🙂