memory vs. server load

Im having issues with my wordpress site. Its become very popular, and with all of this traffic, Im either having too much server load, or too much server memory being used. I am using the WP super cache plugin, and when I use it the memory usage goes way up. If I get rid of it, the memory usage drops down a lot, and my server load goes up big time. Am I doing something wrong or is this basically what Im looking at with a lot of traffic?
I dont have many plugins installed that I think would be causing problems, just akismet, all in one SEO, feedburner, google sitemaps, mighty adsense, search everything, shorturl, statcounter, ultimate google analytics, and WP super cache.