I have spent over a month trying to find the answer to this question. All the forms have people just saying, "I have the same problem" or "it's a problem with your web hosting" but no real way to resolve the issue.
Here is what worked for me, on all my 4 sites. And believe me I tried EVERYTHING. I went from load time of high 9's and 15's (seconds) down to low 1's and 2's (seconds).
Here is what I tried:
1. Install "Quick Cache" Plugin http://wordpress.org/extend/plugins/quick-cache/
2. Change your .htaccess code http://www.improvespeed.info/2010/01/how-to-optimize-wordpress-htaaccess.html
3. which put my site on turbo https://www.cloudflare.com/ ( I have all my sites on Cloud Flare now ) I love the speed. I was hesitant to make DNS changes from my hosting but boy am I glad I did. (I use their free service)
BACKUP BACKUP BACKUP before trying any of these steps "Especial, copy the .htaccess before making changes".
These steps worked for me, I take no responsibility if something goes wrong for you.