I can second this behavior. I am seeing the exact same results. I was at first just trying to figure out why I had such a long server response time lag with W3TC turned on (even with APC enabled). My page load times were great - but the initial server response was unbearable. It was bad in the Admin too, where you wouldn't think W3TC would be an issue.
When I finally disabled W3TC, my server response times were near instantaneous, but the page load of course was much longer. I was getting a consistent 7 second lag on an in-house clean development server, but no lag with the plugin off.
I then found this topic, and tried the hint of turning on debug mode. My lag dropped to 2 seconds (and that last 2, as confirmed in the debug mode results at the bottom of the page, included some plugin queries that took far too long).
This is on WP 2.9.2, on a windows server, with everything up-to-date, a slew of plugins, and a DB consisting of 250 authors, 11,000 posts and 140,000 comments.
I never suspected W3TC, which was making my pages load wicked fast, would be causing the lag. When I was observing our live server statistics in real-time, I could see the CPU never got above 5% usage, and our 8GB of RAM was only hovering at about 30% usage. So it made no sense that the server was taking so long to respond (on the live server, up to 15 seconds) and begin streaming the page data - there was no stress on the server of any kind - just a long pause.