Yea, if you're using wp_nav_menu() you should be able to control the nav output by going to Appearance > Menus in the admin. When you get to the Menus page in the upper right corner click "Screen Options" then check the "CSS Classes" checkbox, which will allow you to assign classes to your menu items. Do something like give all the ones you want to pop up in a modal a similar class, e.g. modal.
Get your favorite javascript library, I prefer jQuery and then snag some simple lightbox/modal plugin, Colorbox is a super solid one, then target the modal objects with your JS, sorta like this:
$(".modal a").colorbox({iframe:true, innerWidth:425, innerHeight:344});