Have you disabled the cache plugins for any length of time to confirm there isnt an issue there?
What about running a repair/optimize on the db itself?
What do the apache logs say right befor the crash?
Hi all,
Thank you for responding – it is much appreciated.
I only put on the caching plugins to try to deal with the problem. On the new Ubuntu 10.04 system, it has never been online with the caching plugins active.
I have optimized the tables with (for example):
OPTIMIZE TABLE wp_users
repeating that for all tables within the wordpress database. If there is something else I should be doing for this, please point me in the right direction and I will investigate.
Unfortunately the apache logs don’t really have anything there. The apache error log is virtually empty (beyond the occasional file not found) and the access log has just GET requests.
The /var/log/messages file has the following repeated all over after the problem starts until I reboot.
Aug 23 02:46:00 www5 kernel: [561085.858680] apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Aug 23 02:46:00 www5 kernel: [561085.858684] apache2 cpuset=/ mems_allowed=0
Aug 23 02:46:00 www5 kernel: [561085.858688] Pid: 20723, comm: apache2 Not tainted 2.6.32-21-generic-pae #32-Ubuntu
Aug 23 02:46:00 www5 kernel: [561085.858690] Call Trace:
Aug 23 02:46:00 www5 kernel: [561085.858698] [<c01d4d34>] oom_kill_process+0xa4/0x2b0
Aug 23 02:46:00 www5 kernel: [561085.858701] [<c01d53a9>] ? select_bad_process+0xa9/0xe0
Aug 23 02:46:00 www5 kernel: [561085.858704] [<c01d5431>] __out_of_memory+0x51/0xa0
Aug 23 02:46:00 www5 kernel: [561085.858706] [<c01d54d8>] out_of_memory+0x58/0xb0
Aug 23 02:46:00 www5 kernel: [561085.858709] [<c01d7d17>] __alloc_pages_slowpath+0x407/0x4a0
Aug 23 02:46:00 www5 kernel: [561085.858712] [<c01d7eea>] __alloc_pages_nodemask+0x13a/0x170
Aug 23 02:46:00 www5 kernel: [561085.858716] [<c01fa8a6>] read_swap_cache_async+0xa6/0xf0
Aug 23 02:46:00 www5 kernel: [561085.858718] [<c01fa957>] swapin_readahead+0x67/0x90
Aug 23 02:46:00 www5 kernel: [561085.858721] [<c01ef2ae>] do_swap_page+0x47e/0x570
Aug 23 02:46:00 www5 kernel: [561085.858723] [<c01ef737>] handle_mm_fault+0x397/0x4a0
Aug 23 02:46:00 www5 kernel: [561085.858727] [<c0213387>] ? __fput+0x177/0x1f0
Aug 23 02:46:00 www5 kernel: [561085.858731] [<c05b3dad>] do_page_fault+0x10d/0x3a0
Aug 23 02:46:00 www5 kernel: [561085.858734] [<c05b1fb0>] ? do_device_not_available+0x0/0x60
Aug 23 02:46:00 www5 kernel: [561085.858736] [<c05b3ca0>] ? do_page_fault+0x0/0x3a0
Aug 23 02:46:00 www5 kernel: [561085.858743] [<c05b1c63>] error_code+0x73/0x80
Aug 23 02:46:00 www5 kernel: [561085.858744] Mem-Info:
Aug 23 02:46:00 www5 kernel: [561085.858746] DMA per-cpu:
Aug 23 02:46:00 www5 kernel: [561085.858748] CPU 0: hi: 0, btch: 1 usd: 0
Aug 23 02:46:00 www5 kernel: [561085.858749] Normal per-cpu:
Aug 23 02:46:00 www5 kernel: [561085.858750] CPU 0: hi: 186, btch: 31 usd: 181
Aug 23 02:46:00 www5 kernel: [561085.858754] active_anon:42505 inactive_anon:42673 isolated_anon:0
Aug 23 02:46:00 www5 kernel: [561085.858755] active_file:205 inactive_file:335 isolated_file:0
Aug 23 02:46:00 www5 kernel: [561085.858755] unevictable:0 dirty:0 writeback:103 unstable:0
Aug 23 02:46:00 www5 kernel: [561085.858756] free:1499 slab_reclaimable:667 slab_unreclaimable:2500
Aug 23 02:46:00 www5 kernel: [561085.858757] mapped:207 shmem:10 pagetables:1878 bounce:0
Aug 23 02:46:00 www5 kernel: [561085.858762] DMA free:2064kB min:84kB low:104kB high:124kB active_anon:2076kB inactive_anon:2688kB active_file:116kB inactiv
Aug 23 02:46:00 www5 kernel: [561085.858766] lowmem_reserve[]: 0 492 492 492
Aug 23 02:46:00 www5 kernel: [561085.858772] Normal free:3932kB min:2792kB low:3488kB high:4188kB active_anon:167944kB inactive_anon:168004kB active_file:70
Aug 23 02:46:00 www5 kernel: [561085.858776] lowmem_reserve[]: 0 0 0 0
Aug 23 02:46:00 www5 kernel: [561085.858779] DMA: 8*4kB 2*8kB 12*16kB 23*32kB 17*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2064kB
Aug 23 02:46:00 www5 kernel: [561085.858785] Normal: 633*4kB 135*8kB 20*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3932kB
Aug 23 02:46:00 www5 kernel: [561085.858792] 11942 total pagecache pages
Aug 23 02:46:00 www5 kernel: [561085.858793] 11388 pages in swap cache
Aug 23 02:46:00 www5 kernel: [561085.858795] Swap cache stats: add 17605162, delete 17593774, find 1247671/3129912
Aug 23 02:46:00 www5 kernel: [561085.858796] Free swap = 0kB
Aug 23 02:46:00 www5 kernel: [561085.858797] Total swap = 397304kB
Aug 23 02:46:00 www5 kernel: [561085.860542] 131056 pages RAM
Aug 23 02:46:00 www5 kernel: [561085.860544] 0 pages HighMem
Aug 23 02:46:00 www5 kernel: [561085.860545] 3896 pages reserved
Aug 23 02:46:00 www5 kernel: [561085.860546] 11980 pages shared
Aug 23 02:46:00 www5 kernel: [561085.860547] 124646 pages non-shared
One more thing…
The site is rather low in traffic. We have MUCH busier sites with no problems at all. I think there has to be more going on here than a pure optimization issue. I just don’t know how to track it down.
From top currently – the system is not unresponsive yet, but it is getting slower and slower. The apache processes just hanging around for more than a minute concern me.
P.S. If you want to look at the site directly it is at:
www5.newpaltz.edu/little-rebellion
top - 11:11:54 up 1:23, 4 users, load average: 3.41, 2.24, 1.14
Tasks: 87 total, 5 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 95.7%us, 3.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 508876k total, 445768k used, 63108k free, 37160k buffers
Swap: 397304k total, 212k used, 397092k free, 147572k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1259 www-data 20 0 47868 22m 4044 R 14.3 4.4 1:06.96 apache2
1458 www-data 20 0 49148 22m 3532 R 13.3 4.5 1:00.90 apache2
1459 www-data 20 0 46816 20m 4048 R 10.7 4.2 1:01.53 apache2
1276 www-data 20 0 46732 20m 3516 S 9.3 4.1 1:06.40 apache2
1278 www-data 20 0 49152 23m 4092 R 9.0 4.6 1:05.39 apache2
1282 www-data 20 0 47780 21m 3532 S 9.0 4.3 1:06.08 apache2
1258 www-data 20 0 48404 22m 3660 S 8.7 4.5 1:06.12 apache2
1260 www-data 20 0 53444 27m 4324 S 8.3 5.6 1:06.96 apache2
1455 www-data 20 0 46772 20m 3512 S 8.3 4.1 1:03.84 apache2
1262 www-data 20 0 47752 21m 4048 S 8.0 4.4 1:06.56 apache2
692 mysql 20 0 144m 49m 5988 S 0.7 9.9 0:09.52 mysqld
1 root 20 0 2652 1476 1196 S 0.0 0.3 0:00.71 init
We are having the exact same issue. Has anyone found a resolution yet?