If you have mod_rewrite selected and caching enabled, and you then change mode or turn caching off the .htaccess rules aren't removed.
This means the caching is still enabled despite the plugin telling the user that it is not.
This behaviour is non-intuitive.
Deleting the rules when un-selecting caching/mod_rewrite would make sense.