I wanted to try Quick Cache on a more static site of mine, and so I deactivated WP Super Cache, removed the rewrite rules, installed Quick Cache and turned it on. Worked fine.
Then I wanted to uninstall WP Super Cache. This plugin seems to do a lot of stuff when being uninstalled (or if it is Word-press itself, dunno), like deleting advanced-cache.php and such.
But the real problem is that it uncommented define('WP-CACHE') in wp-config.php, and while doing so it also removed the initial <?php tag.
This means that what is displayed when any page on the site is called is the wp-config-php file, in plain text, with passwords to the DB and all, for everyone to read.
If this had been on a more active site, I could have been seriously screwed by now.