I think all plugins will cause decreasing of website perfomance. The important thing is how much the slowliness compare to the features you get.
I personally experinced high CPU load if I enable the "File Change Detection" feature in this plugin. Other than that, I never seen any significant perfomance issue.
My suggestion is only enable the features you understand and need. Also as being mentioned by mbresolution, you should clean your log file frequently.
I've been working hard on speeding up my website, ...
You may need to consider to move to a better webhosting. Previously, no matter how much optimizations I've done, my website was always slow. After moved to a good webhosting, now my sites can use up to 1 MB memory and the speed are above average.