If you are seeing a GMT-4 in your wordpress settings, you are probably not running wp 2.8, AND do not have the automatic timezone plugin enabled.
In that case, the plugin attempt to guess the more accurate wordpress timezone, by assuming a default city in your case '-4'=>'America/Manaus', See in the plugin settings what the pklugin reflects as your wp timezone. Fix by upgrading to 2.8 or enabling automatic timezone plugin.
You say the plugin is reading the ics timezone file as Etc/GMT+4 - That must then be what the ics file says! - Have you checked the .ics file directly?
Times are then adjusted depending on the timezone specified in the ical events - there are many ways of specifying these. So far there have not been any error in time calcs, just in peoples understandings of their own timezone settings and the ics file settings.
The plugin does not read the server timezone - is the server the one that is producing the ics file? Perhaps that is what is causing the confusion?
If you send/post your ics file link, and the page that you are attempting to use it on, I can check it, or you can check against the test site. See http://anmari.com/testing/wp/?page_id=127