Heavy load with 40000 Unique Visitors

I am not sure what the heaviest trafficed wordpress site is ( though I would be interested to find out ), I run a site that receives 40,000 Unique Visitors a day. I have been runnning it on a dedicated server running RHEL4, MySQL 4.1, Apache 2, two 2.4 ghz intels, 4 gigs of RAM. We regularly reach 500 visiters online (using wp_onlinecounter), with 697 being our max.

Our main site spikes periodically through out the day regularly reaching up to 54 load avg. No matter how high I set the Apache MaxClients, it extends to “2” passed the limit. Right now I have it set to 300 and I reach 302 and start hitting swap. (I set it to 500 once. BIG MISTAKE – CPU load hit 154+ and crashed my server). Normally, we don’t hit swap.

Accourding to MyTop, I see a huge amount of “Sleeping” queries and which suggest to me that WordPress is not closing the connections (perhaps the reason with the spikes).

I have turned off most of the plugins. I recently installed WP-Cache but have yet to see the results. I also updated to WordPress 2.0.5 and install “Referer Karma” plugin to watch for bots.

Unforunately, we need to use .htaccess REWRITE rules and Permalinks because my boss loves the SEO and the ease of use.

With Apache, I have KeepAlive on with 100 max requests and 15 sec time outs.

With MySQL, well its better if I just include it.

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

log-slow-queries=/var/log/mysql_slow_query.log
open-files-limit=1000

query_cache_type=1
query_cache_limit=2M
query_cache_size=64M

key_buffer=1024M

set-variable = max_connections=3072
set-variable = thread_cache=40
set-variable = back_log=500
set-variable = table_cache=256M
set-variable = read_rnd_buffer_size=3M

set-variable = max_allowed_packet=1M
set-variable = max_connect_errors=999999

skip-locking
skip-name-resolve

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[client]
socket=/var/lib/mysql/mysql.sock

I will not switch to pconnect because I am using only one local server and pconnect will only make matter worse.

I was thinking of explitely hacking the core files to add a mysql_close function to the db object and called it a the very end of the footer.php. That will only save a few milliseconds (if any).

I can’t find anything else to improve performance and prevent the periodic spikes. I am beating my head against the wall to get more out of this server and optimize my software (including WordPress).

Any ideas? Is this normal for wordpress to spike the CPU like this?

What is the most heavily trafficed word press site anyways? Perhaps, they can give me some hints.

Any help would be appreciated. Thanks, cmack AT madmanmedia DOT com