The shadowbox.css is loaded in the header section, so before any post is in the_loop.
As of the plugin is based on a filter parsing the_content of an blog post, it cannot know if there is any Shadowbox on a page – even if <object> is choosen as default display option, the [flash] tag value mode=3 would still be a possible way to embed flash via Shadowbox.
So I think the “dirty” way by forcing shadowbox.css (to ensure a “mode=3” would still work) is acceptable. The file is not that big, any user should be able to load it once (should stay in cache).
I was just looking into this myself. It would be great having an option to totally disable Shadowbox, including the CSS file. I know the file is small, but many plugins want to load their own files these days, and the total number of requests really adds up.