Site loads slow, stop process, reload

My site at times loads very slow. If I stop the loading of the page by pressing x and then refresh page it loads immediately. What could be causing this?

I am on a VPS server that has 4GB Ram is Quad Processor and I only have two sites on my VPS server.

I understand that plugins can slow down the loading but why is it that if the loading of the site is stopped and then refreshed it loads immediately?