Well, I'm going to assume it's some flavor of *NIX and Apache/HTTPD you're running. I struggled with the same problem for the first few months. I don't know you're exact setup, but I used an inexpensive Rackspace virtual/cloud server with only 512MB RAM. Now, it's been fine for a long time using Apache Virtual Server on one IP with numerous fast loading Web sites - that is until I threw a WordPress site on it.
The other sites were fast as hell but my WordPress blog would take a horrifically long 10 - 20 seconds to load each page. Did endless searching. I ruled out the server and MySQL as culprits. I finally came across a post that went over tuning the entire LAMP stack. What did the trick finally for me was to tune my httpd.conf for WordPress - again, although the 6 other sites on the same instance were all super speedy. Here's what I changed in my httpd.conf:
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 50
MaxClients 50
MaxRequestsPerChild 4000
</IfModule>
<IfModule worker.c>
StartServers 5
MaxClients 300
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 2000
</IfModule>
KeepAlive Off
Every situation/server will be slightly different, but this worked for me and completely resolved my slow WordPress issues (after restarting Apache). I also signed up for CloudFare Content Delivery Network (CDN). That seems to have contributed to a slight speed increase as well.
Good luck.