What other plugins are you running. Realize that chat makes AJAX calls to your server via the standard WordPress AJAX hooks. The down side here is for every AJAX call it means WordPress has to be initialized. So if your Chat AJAX interval is set to one second it means this is equal to reloading the page every second for every user participating in chat.
Also, you did not specify how many users were participating in chat. Was it just you?
Lastly, to say you have a beefy dedicated server really does not mean it it properly tuned. I know most hosing companies assume customers who order a dedicated server know what they are doing. So MySQL and Apache are pretty much left out of the box untuned. This may not be the case in your instance. But you might want to double check your MySQL configuration and make sure you at least have MySQL cache enabled with enough memory.