There are a lot of variables that can play into that. There are a few places in my plugin that I know are performance issues, but I don't know how to fix them. They all stem from the same problem.
When a user sets up a secure site on a subdomain, I change it so that the cookies are set for the base host. That way if you have example.com and secure.example.com, you will be logged into both.
There is no good way to figure out the base host of a random URL. The only way I found to do it was to strip subdomains off of the host until you hit something that's not a valid domain.
secure.example.com - good
example.com - good
com - bad
So, example.com is the base domain. In order to test if a domain is valid, you have to make a request from the server to that domain. This generally isn't a problem, but has the potential to slow down the page when you login, update the plugin settings, or update the plugin. These are all the places where those kinds of calls are made.
Edit: Just to clarify, I'm not saying my plugin is the cause, but it's possible. HTTPS is always going to be slower because of the encryption/decryption process. Like I said before, there are also other factors that can play into it.