I have WordPress installed on a dedicated box with 1GB of RAM and WordPress is using 700MB of it, maxing out the available RAM, then heading over to swap for ANOTHER 1GB and it still wanted more
The site is only at 10k uniques per day on average
I've tried the site on 2 different dedicated machines at 2 different hosts and both had RAM/Swap issues. The hosts think it MAY be something with WordPress and the Debian libraries. Whatever that means.
I even installed WP-Cache to try to resolve it and there was absolutely 0 effect