This actually isn't due to the plugin but the downloading of data from Google Analytics for your stats. The more traffic you get the more data it has to download. There are a few ways to help it.
1. Only have it display the last 7 days so it's not downloading data from the last 30 days.
2. Change your "Cache Timeout" in the options. Right now it is set to 60 seconds. That means every minute it is going to download data from Google. Unforunately, there is no way with their API to download partial data. Really, you can only download all the data each time. So, if you set the cache timeout to download every 3600 seconds it will only download the data hourly instead of every minute.
3. Disable the page/post stats. When it downloads the data for post and pages it has to download the data for each page and post. If you have a lot of post or get a lot of traffic this could take a while for the site to download all that data. If you don't want to wait for the data to download you could disable the stats on the post and page admin pages.
I hope this helps some.