I have struggled with loads for a few months, and I too both own my own servers and host on VPSs for all my wordpress blog installations. However all my servers have been Linux varieties.
wp-cache does help a lot, but there are still problems with how loads are carried on the servers when using wordpress.
I have one wordpress installation that previously was on a VPS that was not on a server I owned and it has about 20 different plugins working: it had massive load issues that disappeared when wp-cache was added. After about 3 months of using wp-cache, it is still trouble-free :)
But, I have another wordpress installation that has only about 3-4 plugins active and was also having massive load issues on a VPS. We used wp-cache, but that didn't help much. I then moved it to my own dedicated server and my technician still saw major problems in how mysql was being queried. Odd, but it is still having major problems and I just moved it to Laughing Squid hosting since others on this forum say they should be able to handle WordPress hosting. I made the move a few days ago and will know in about a week if my problems have disappeared.
Granted, many other software out there like drupal, phpnuke, xoops and others do have similar mysql query problems, but since I'm working with wordpress and not those others, I'd like to see how wordpress' core functions can best be optimized in the future :)