I have on my site other files and directories outside of WordPress. Some of files have links that go to different cases of the files. So I found a solution that was to add the following to my .htaccess file to make the URL's case insensitive.
This works by itself. The problem is, it doesn't play nice with the mod_rewrite section added by WordPress. If I remove the WordPress code from the htaccess it works but WordPress permalinks don't.