Not sure what is happening. My hostmaster just upgraded his systems, so I'm not sure if this is your plugin or not since more than one thing seemed to changed (also recently upgraded to WordPress 3.1.3). When the widget or page with a calendar listing is included in my WordPress setup, it takes at least 30 seconds or more for it to come up. I thought it might be the Google calendar, but when I put the calendar URL in directly or go to the page with the Google calendar iframe included, it comes up immediately (relatively). When I take out your widget call, my pages come up very quickly. Then I tested the page that is supposed to show the events, and it also takes a very long time.
The URL after I took out the widget is http://www.uulosalamos.org and the URL with the events page is http://www.uulosalamos.org/happenings/
Maybe I have a badly-formatted ics URL? I can send you those if needed.