I can see both sides of the argument here, but after reading this thread -- I have to say I fully agree with Justin Tadlock. It should be up to plugin authors to utilize the WP hooks appropriately.
If the plugin author writes a crappy plugin, you shouldn't be using it regardless of whether you can control what page(s) it loads on or not. If their implementation is crappy, their code is probably crappy, and if their code is crappy, there's a good chance it poses a security vulnerability as well.
Besides, a lot of plugins have API endpoints that are used and never documented, and good luck getting support from many of these plugin authors to even know what these endpoints are so you can ensure the plugin is loaded when those endpoints are hit from an external application. For example, think of any eCommerce plugin that receives an IPN notification from PayPal, or a Webhook from Stripe, or a Silent Post from Authorize.net.
I think a feature like this doesn't make sense at the WP core level, and would cause far more headaches than it would solve -- not to mention needless bloat to an already large (and ever growing) code-base.
There are a few plugins that do this, and that is the great thing about WordPress; that it is extensible enough for a 3rd party developer to do something like this. Besides, a plugin that can do this is, in my opinion, more than sufficient for those folks who are truly concerned about performance.