[Plugin: WooCommerce] – Checking Out with Cash on Delivery is Slow

Using wordpress 3.6.1
Woocommerce 2.0.14

I see that many people had this issue where by their checkout process falls into an endless loop. And there seems to be some fix here

http://docs.woothemes.com/document/endless-loadingspinner-on-the-checkout-page/

But mine is slightly different whereby I can checkout with paypal and it will redirect to paypal site in like less then 10s.

The problem is when I try to use Bank Transfer, Cash on Delivery or Check payment, it just sits there and will take like 2 minutes to reach the Order Received page

Have set:
1) 128Mb of memory limit
2) Woocommerce System Status is good
3) WordPress address and siteurl is the same
4) Deactivating all other plugin except woocommerce
5) using theme twentytwelve and twentythirteen
6) even tried having a fresh install wordpress + woocommerce plugin but still slow

all seem to be the same. I wonder if there is something wrong the ajax for woocommerce-checkout process or if something configure to be not efficient on my VPS server