Not as easily, no. The reason for this is:
-Single posts can belong to multiple categories, so which category template should it use?
-Archive pages can have multiple posts, so again, which category template should it use?
That said, it would be possible to roll your own code to deal with this sort of thing.
All templates get run through a filter. In your case, the filters you're interested in are single_template and archive_template. You could do something like this:
// $template contains the filename of the template being used (like "/single.php")
// decide what template you want to use
$template = "/single-whatever.php";
// send it back
Basically, the filter gets the filename of the template. You can change it and return it and thus force a different template. You would probably want to put this sort of code in your theme's functions.php file.