Hello dhobeika. Yes, your initial load is slow.
First, I would check an plugins. Any plugin that refers to another site or is coded to login to another site may be causing your problem. Try to remove highly probably plugins first.
Another thing I try is to reinstall my database. Once in a while, I will create a new database. Do a backup and then restore to the new db. Change my wp-config. php file. Then I throw away the old database (once I've checked things out.) I also know there is a plugin to clean up the database. I don't remember what it is, but your database accumulates junk, just like your physical computer drive does.
Here is another link that might give you more information.
http://bit.ly/js5rgW. That's not my website, but it is one that gives me good tips.