Another user first complained about slowness 6 months ago, citing the P3 Plugin Profiler, and I've made several improvements to the plugin since then. However, I've never been able to reproduce the slowness that others have complained about. (The numbers I get with P3 put my plugin about average with other plugins I regularly use.) But there are other variables of different server settings, themes, active plugins, etc.
Did you check "Disable Gravatar and use only local avatars"? The plugin uses object caching if you still want to use Gravatar, but I think the calls to Gravatar is one of the culprits of slowness.
Also, for reasons that are still a mystery to me, sometimes simply disabling and re-enabling the plugin makes it work better.
I also suggest temporarily switching your theme to the Twenty Twelve theme to see if there is a problem with your theme. I understand that not every site has the luxury of doing this, but I always test my plugin with Twenty Twelve.