I was just looking at the code, and see that -all- active plugins are loaded every time WordPress runs. This means that certain plugins only used when the admin is accessing the back-end are also being loaded for every page load. Some examples:
Deactivate Visual Editor
It seems that there should be a way for plugin authors to specify that it's an admin-only plugin, so get_option('active_plugins'); doesn't return it. Perhaps there could be a get_option('menu_plugins') which is checked on admin page loads.
At the same time, I would suggest having a get_option('wpmu_plugin') queue as well (the code would be very similar). There are plugins that I'd like to make available only to is_site_admin(), but putting it in mu-plugins only means it's 'activated' by default for every page load. It has nothing to do with whether the plugin is for the wpmu admin or not. Plugin Commander would be a great candidate for this.
This is related to the "Make It Faster"
http://wordpress.org/extend/ideas/topic.php?id=127 Right now, every active plugin slows down WordPress, regardless of whether it's used or not.