Never assume that it is safe to delete the files and directories in /wp-content/cache/. WP Super Cache recursively scans the /wp-content/cache/ directory and deletes anything it finds. This is NOT okay. WP Super Cache's behavior during network deactivation caused a disaster yesterday because there were custom symlinks in /wp-content/cache/ms-filemap/ and WP Super Cache stupidly decided to walk that subdirectory, wiping out every single file in there. Each symlink is mapped to /wp-content/blogs.dir/[blognum]/. Fortunately, we have pretty good backups, so the loss was minimal - just a few images. Without the backups, we would have lost pretty much everything thanks to this bug.
Also, it looks like the deactivation process deletes the configuration. Please don't do that either - at least not without making a backup. Then, if someone activates the plugin again, look for a backed up config from a previous activation.
Deactivation of all plugins should be near-instantaneous. Deleting the entire cache typically exceeds the length of time most servers allow scripts to run. If you want to allow someone to delete files on deactivation, let it be an option within the plugin itself that can be enabled (disabled by default).