Apache is not using SWAP and it's reporting page allocation failure

Question

Check the logs:

    httpd: page allocation failure. order:4, mode:0xd0

Call Trace:
[<ffffffff8000f696>] __alloc_pages+0x2ef/0x308
[<ffffffff80017a40>] cache_grow+0x139/0x3c7
[<ffffffff8005bd54>] cache_alloc_refill+0x138/0x188
[<ffffffff800df358>] __kmalloc+0x95/0x9f
[<ffffffff800f0bba>] alloc_fd_array+0x18/0x28
[<ffffffff800306c8>] expand_files+0x124/0x2cb
[<ffffffff800226c5>] dup_fd+0x133/0x281
[<ffffffff80049050>] copy_files+0x47/0x64
[<ffffffff8001f632>] copy_process+0x590/0x15db
[<ffffffff800a11c4>] alloc_pid+0x228/0x2c4
[<ffffffff80030f6c>] do_fork+0x69/0x1c1
[<ffffffff8005d116>] system_call+0x7e/0x83
[<ffffffff8005d427>] ptregscall_common+0x67/0xac

Mem-info:
Node 0 DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
cpu 6 hot: high 0, batch 1 used:0
cpu 6 cold: high 0, batch 1 used:0
cpu 7 hot: high 0, batch 1 used:0
cpu 7 cold: high 0, batch 1 used:0
Node 0 DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:14
cpu 0 cold: high 62, batch 15 used:57
cpu 1 hot: high 186, batch 31 used:29
cpu 1 cold: high 62, batch 15 used:58
cpu 2 hot: high 186, batch 31 used:30
cpu 2 cold: high 62, batch 15 used:57
cpu 3 hot: high 186, batch 31 used:30
cpu 3 cold: high 62, batch 15 used:56
cpu 4 hot: high 186, batch 31 used:26
cpu 4 cold: high 62, batch 15 used:13
cpu 5 hot: high 186, batch 31 used:8
cpu 5 cold: high 62, batch 15 used:42
cpu 6 hot: high 186, batch 31 used:30
cpu 6 cold: high 62, batch 15 used:50
cpu 7 hot: high 186, batch 31 used:157
cpu 7 cold: high 62, batch 15 used:59
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:13
cpu 0 cold: high 62, batch 15 used:50
cpu 1 hot: high 186, batch 31 used:66
cpu 1 cold: high 62, batch 15 used:49
cpu 2 hot: high 186, batch 31 used:3
cpu 2 cold: high 62, batch 15 used:55
cpu 3 hot: high 186, batch 31 used:104
cpu 3 cold: high 62, batch 15 used:55
cpu 4 hot: high 186, batch 31 used:2
cpu 4 cold: high 62, batch 15 used:2
cpu 5 hot: high 186, batch 31 used:87
cpu 5 cold: high 62, batch 15 used:3
cpu 6 hot: high 186, batch 31 used:15
cpu 6 cold: high 62, batch 15 used:55
cpu 7 hot: high 186, batch 31 used:113
cpu 7 cold: high 62, batch 15 used:12
Node 0 HighMem per-cpu: empty
Free pages: 76972kB (0kB HighMem)
Active:3061605 inactive:620399 dirty:68055 writeback:0 unstable:0 free:19243 slab:283635 mapped-file:33024 mapped-anon:2758457 pagetables:91114
Node 0 DMA free:10032kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:9628kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3451 16066 16066
Node 0 DMA32 free:53992kB min:3484kB low:4352kB high:5224kB active:2704704kB inactive:367984kB present:3534588kB pages_scanned:268 all_unreclaimable? no
lowmem_reserve[]: 0 0 12615 12615
Node 0 Normal free:12948kB min:12732kB low:15912kB high:19096kB active:9541756kB inactive:2113612kB present:12917900kB pages_scanned:423 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 4*4kB 2*8kB 3*16kB 3*32kB 2*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10032kB
Node 0 DMA32: 0*4kB 59*8kB 45*16kB 82*32kB 0*64kB 6*128kB 5*256kB 0*512kB 1*1024kB 1*2048kB 11*4096kB = 53992kB
Node 0 Normal: 47*4kB 187*8kB 356*16kB 148*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 12948kB
Node 0 HighMem: empty
923737 pagecache pages
Swap cache: add 180089, delete 179928, find 2859/3518, race 0+0
Free swap = 8304596kB
Total swap = 8388600kB
Free swap: 8304596kB
4322816 pages of RAM
230069 reserved pages
14241581 pages shared
117 pages swap cached

My /etc/fstab (SWAP is a file on an SSD):

proc /proc proc defaults 0 0
none /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm        tmpfs   defaults        0       0
/dev/md0 /boot ext3           defaults 0 0
/dev/md1        /       ext4    rw,noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,usrquota,grpquota,jqfmt=vfsv0       0       0
/dev/md2        /home   ext4    rw,noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,usrquota,grpquota,jqfmt=vfsv0       0       0
UUID=b422827c-9f2f-4bc5-ab5a-3cc921924daa /var/lib/mysql ext4 rw,noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,usrquota,grpquota,jqfmt=vfsv0,  1      1
/var/lib/mysql/swapfile1 swap swap defaults 0 0

Swapiness is set to 20:

cat /proc/sys/vm/swappiness
20

There is only a few RAM left and still using very low SWAP till it crashes and the RAM is freeded and I got another crash in a few hours:

Mem:  16370988k total, 16200724k used,   170264k free,   250340k buffers
Swap:  8388600k total,    83356k used,  8305244k free,   859484k cached

Answer 1

What is /proc//oom_adj ?

How does the memory demand change - a sudden increase might indicate you are hitting the 2Gb limit. A slow increase implies you've definitely got maxclients set too high, and proabably need more hardware. It makes little sense to use big iron as a webserver - use several smaller boxes.

What kernel version - see this bug report (and others).