wow, this is a highly unusual request.
First, note that CSS gets cached. So, despite it being 'irrelevant' to other pages/posts, having it be in one file makes many things easier (only one to download ever, always the same one...).
Second, the 'conditional' solution can be achieved a few ways. By the time you reach your header.php, the query has run. $p >might< be set to the post Id. if not, the query should have detected is_single, and will be able to give you the post id.
With the post id retrieved into a variable at the top of say header.php, there's a few approaches:
1. if the post ID matches the custom one, include the custom css instead of the standard css. that's pretty simple. You could completely change the style, though the basic components of the theme remain.
2. set the ID of the body or first div to "post-".$postID (where $postID is the thing you grab at the top). In the CSS, you can then have targeted overrides for just that post. But again, then only really need one CSS file. Overhead should be minimal.
3. write a custom plugin that pulls up a custom theme when that single post is the query result. then you can modify both theme AND css to your complete needs.