The current implementation is really short and sweet; a single line regexp finds all links-to-images that doesn't already have a rel-attribute, adds the rel="lighbox" (grouping them by post ID).
To separate a post's galleries from eachother we'd need to add the rel attribute long before this step - somewhere where WordPress is still dealing with the gallery-shortcode (and thus aware of their attributes).
A first line of attack might be;
1. filter post_gallery to read our new shortcode attribute ("instance" or "set")
2. put the current set ID in a global variable to make it available to...
3. ... a wp_get_attachment_image_attributes-filter, where we now can add rel="lightbox[#set]" to the output.
Seems like it should work without much overhead. We'll just bail if post_gallery does not provide the set-attribute.
One could also forego the shortcode attribute and make this behaviour a global setting - "always group by instance" or "always group by post".
Convenient, but less flexible.