I have been adding rules and restarting shorewall for a couple years now on this box (3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.39-1+deb8u1~bpo70+1).
Shorewall version 4.5.5.3
iptables v1.4.14
Mem: 3.8G 2.4G 1.4G 0B 67M 359M
-/+ buffers/cache: 2.0G 1.8G
Swap: 9G 338M 9.7G
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 3054.953
BogoMIPS: 5599.87
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 10240K
NUMA node0 CPU(s): 0-7
I recently added 3 lines in shorewall config and restarted shorewall but the server crashed. Upon reboot and several retries we get this in the kern.log.
I went back, thinking this might be iptables entries are getting too many, removed a good 150 entries and tried again with the same result.
What i want to understand is if this is an issue with the RAM and if i add another 4g-8g of RAM this will be resolved or if this might be a memory leak issue or anything else with iptables.
I can add more RAM, i can even change hardware if needed, what i cannot do currently is upgrade anything on the system software-wise since there are too many moving parts and these will get replaced in about 6-8 months.
kernel: [ 57.165772] iptables invoked oom-killer: gfp_mask=0x2d2, order=0, oom_score_adj=0
kernel: [ 57.165776] iptables cpuset=/ mems_allowed=0
kernel: [ 57.165779] CPU: 5 PID: 15056 Comm: iptables Tainted: G O 3.16.0-0.bpo.4-amd64 #1 Debian 3.16.39-1+deb8u1~bpo70+1
kernel: [ 57.165781] Hardware name: Dell Inc. PowerEdge R320/08VT7V, BIOS 2.1.2 01/20/2014
kernel: [ 57.165782] 0000000000000286 0000000000000000 ffffffff8154f97f 0000000000000007
kernel: [ 57.165784] ffff8800a218b3f0 0000000000000000 ffffffff8154cdf7 0000000000000000
kernel: [ 57.165786] 0000000000000002 0000000000000007 ffffffff810cec20 00000000ffffffff
kernel: [ 57.165788] Call Trace:
kernel: [ 57.165795] [<ffffffff8154f97f>] ? dump_stack+0x5e/0x7a
kernel: [ 57.165797] [<ffffffff8154cdf7>] ? dump_header+0x76/0x1ec
kernel: [ 57.165800] [<ffffffff810cec20>] ? rcu_batches_completed+0x10/0x10
kernel: [ 57.165803] [<ffffffff810e2aef>] ? smp_call_function_single+0x5f/0xa0
kernel: [ 57.165806] [<ffffffff815545de>] ? mutex_lock+0xe/0x2a
kernel: [ 57.165809] [<ffffffff8106f2b7>] ? put_online_cpus+0x27/0x90
kernel: [ 57.165811] [<ffffffff810cfe0c>] ? rcu_oom_notify+0xcc/0xe0
kernel: [ 57.165814] [<ffffffff811534ca>] ? oom_kill_process+0x28a/0x3e0
kernel: [ 57.165816] [<ffffffff81152fbc>] ? find_lock_task_mm+0x4c/0xa0
kernel: [ 57.165818] [<ffffffff81078315>] ? has_ns_capability_noaudit+0x15/0x20
kernel: [ 57.165820] [<ffffffff81153bf4>] ? out_of_memory+0x404/0x550
kernel: [ 57.165822] [<ffffffff8115a065>] ? __alloc_pages_nodemask+0xb95/0xbf0
kernel: [ 57.165826] [<ffffffff8118cb9b>] ? alloc_vmap_area+0x26b/0x300
kernel: [ 57.165828] [<ffffffff8118e939>] ? __vmalloc_node_range+0x129/0x250
kernel: [ 57.165838] [<ffffffffa045fa70>] ? xt_alloc_table_info+0xd0/0x102 [x_tables]
kernel: [ 57.165840] [<ffffffff8118ea91>] ? __vmalloc_node+0x31/0x40
kernel: [ 57.165842] [<ffffffffa045fa70>] ? xt_alloc_table_info+0xd0/0x102 [x_tables]
kernel: [ 57.165844] [<ffffffff8118ec0a>] ? vmalloc_node+0x2a/0x30
kernel: [ 57.165847] [<ffffffffa045fa70>] ? xt_alloc_table_info+0xd0/0x102 [x_tables]
kernel: [ 57.165850] [<ffffffffa046d8f0>] ? do_ipt_set_ctl+0x130/0x200 [ip_tables]
kernel: [ 57.165854] [<ffffffff8148ba9e>] ? nf_sockopt+0xce/0xd0
kernel: [ 57.165857] [<ffffffff810147c0>] ? __switch_to+0x2e0/0x5e0
kernel: [ 57.165858] [<ffffffff8148badf>] ? nf_setsockopt+0x1f/0x30
kernel: [ 57.165861] [<ffffffff8149a14f>] ? ip_setsockopt+0xaf/0xd0
kernel: [ 57.165863] [<ffffffff8143a5be>] ? sockfd_lookup_light+0x1e/0xa0
kernel: [ 57.165865] [<ffffffff815521ee>] ? __schedule+0x2de/0x740
kernel: [ 57.165867] [<ffffffff8143e535>] ? SyS_setsockopt+0x85/0xf0
kernel: [ 57.165869] [<ffffffff81555fcd>] ? system_call_fast_compare_end+0x10/0x15
kernel: [ 57.165870] Mem-Info:
kernel: [ 57.165871] Node 0 DMA per-cpu:
kernel: [ 57.165873] CPU 0: hi: 0, btch: 1 usd: 0
kernel: [ 57.165874] CPU 1: hi: 0, btch: 1 usd: 0
kernel: [ 57.165875] CPU 2: hi: 0, btch: 1 usd: 0
kernel: [ 57.165876] CPU 3: hi: 0, btch: 1 usd: 0
kernel: [ 57.165877] CPU 4: hi: 0, btch: 1 usd: 0
kernel: [ 57.165878] CPU 5: hi: 0, btch: 1 usd: 0
kernel: [ 57.165879] CPU 6: hi: 0, btch: 1 usd: 0
kernel: [ 57.165879] CPU 7: hi: 0, btch: 1 usd: 0
kernel: [ 57.165880] Node 0 DMA32 per-cpu:
kernel: [ 57.165882] CPU 0: hi: 186, btch: 31 usd: 0
kernel: [ 57.165883] CPU 1: hi: 186, btch: 31 usd: 0
kernel: [ 57.165884] CPU 2: hi: 186, btch: 31 usd: 0
kernel: [ 57.165884] CPU 3: hi: 186, btch: 31 usd: 3
kernel: [ 57.165885] CPU 4: hi: 186, btch: 31 usd: 0
kernel: [ 57.165886] CPU 5: hi: 186, btch: 31 usd: 0
kernel: [ 57.165887] CPU 6: hi: 186, btch: 31 usd: 0
kernel: [ 57.165888] CPU 7: hi: 186, btch: 31 usd: 0
kernel: [ 57.165889] Node 0 Normal per-cpu:
kernel: [ 57.165890] CPU 0: hi: 186, btch: 31 usd: 0
kernel: [ 57.165891] CPU 1: hi: 186, btch: 31 usd: 0
kernel: [ 57.165892] CPU 2: hi: 186, btch: 31 usd: 0
kernel: [ 57.165893] CPU 3: hi: 186, btch: 31 usd: 0
kernel: [ 57.165894] CPU 4: hi: 186, btch: 31 usd: 0
kernel: [ 57.165895] CPU 5: hi: 186, btch: 31 usd: 0
kernel: [ 57.165896] CPU 6: hi: 186, btch: 31 usd: 0
kernel: [ 57.165897] CPU 7: hi: 186, btch: 31 usd: 0
kernel: [ 57.165899] active_anon:23913 inactive_anon:5926 isolated_anon:0
kernel: [ 57.165899] active_file:3335 inactive_file:2523 isolated_file:0
kernel: [ 57.165899] unevictable:706 dirty:5 writeback:6083 unstable:0
kernel: [ 57.165899] free:21440 slab_reclaimable:3255 slab_unreclaimable:15149
kernel: [ 57.165899] mapped:5633 shmem:375 pagetables:2459 bounce:0
kernel: [ 57.165899] free_cma:0
kernel: [ 57.165902] Node 0 DMA free:15848kB min:268kB low:332kB high:400kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB
managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? ye
s
kernel: [ 57.165905] lowmem_reserve[]: 0 3191 3896 3896
kernel: [ 57.165907] Node 0 DMA32 free:57876kB min:55148kB low:68932kB high:82720kB active_anon:90840kB inactive_anon:18260kB active_file:10416kB inactive_file:8136kB unevictable:2236kB isolated(anon):0kB isolated
(file):0kB present:3345344kB managed:3270632kB mlocked:2236kB dirty:16kB writeback:18728kB mapped:17716kB shmem:1260kB slab_reclaimable:9868kB slab_unreclaimable:42472kB kernel_stack:3792kB pagetables:7600kB unstable:0kB bounce:0kB free_cma:0
kB writeback_tmp:0kB pages_scanned:194098 all_unreclaimable? yes
kernel: [ 57.165910] lowmem_reserve[]: 0 0 704 704
kernel: [ 57.165912] Node 0 Normal free:12036kB min:12164kB low:15204kB high:18244kB active_anon:4812kB inactive_anon:5444kB active_file:2924kB inactive_file:1956kB unevictable:588kB isolated(anon):0kB isolated(fi
le):0kB present:786432kB managed:721092kB mlocked:588kB dirty:4kB writeback:5604kB mapped:4816kB shmem:240kB slab_reclaimable:3148kB slab_unreclaimable:18124kB kernel_stack:2752kB pagetables:2236kB unstable:0kB bounce:0kB free_cma:0kB writeba
ck_tmp:0kB pages_scanned:22816 all_unreclaimable? yes
kernel: [ 57.165915] lowmem_reserve[]: 0 0 0 0
kernel: [ 57.165917] Node 0 DMA: 2*4kB (UE) 2*8kB (UE) 1*16kB (E) 2*32kB (UE) 2*64kB (UE) 2*128kB (UE) 2*256kB (UE) 1*512kB (E) 2*1024kB (UE) 2*2048kB (ER) 2*4096kB (M) = 15848kB
kernel: [ 57.165926] Node 0 DMA32: 1783*4kB (EM) 1261*8kB (UEM) 746*16kB (UEM) 356*32kB (UEM) 131*64kB (UEM) 35*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 59044kB
kernel: [ 57.165933] Node 0 Normal: 1061*4kB (UEM) 516*8kB (EM) 29*16kB (UMR) 6*32kB (UMR) 2*64kB (UR) 4*128kB (UR) 0*256kB 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 13252kB
kernel: [ 57.165941] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
kernel: [ 57.165942] 13475 total pagecache pages
kernel: [ 57.165943] 6704 pages in swap cache
kernel: [ 57.165944] Swap cache stats: add 36789, delete 30085, find 2817/3620
kernel: [ 57.165944] Free swap = 10360196kB
kernel: [ 57.165945] Total swap = 10476408kB
kernel: [ 57.165946] 1036940 pages RAM
kernel: [ 57.165947] 0 pages HighMem/MovableOnly
kernel: [ 57.165947] 16335 pages reserved
kernel: [ 57.165948] 0 pages hwpoisoned
kernel: [ 57.165949] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
kernel: [ 57.165953] [ 348] 0 348 10074 459 24 228 -1000 udevd
kernel: [ 57.165956] [ 1857] 0 1857 1049 294 8 30 0 ntpdate
kernel: [ 57.165957] [ 1954] 0 1954 1049 0 7 27 0 ntpdate
kernel: [ 57.165959] [ 1956] 0 1956 1538 87 9 22 0 lockfile-create
kernel: [ 57.165960] [ 2001] 0 2001 1049 0 8 26 0 ntpdate
kernel: [ 57.165961] [ 2003] 0 2003 1538 94 9 22 0 lockfile-create
kernel: [ 57.165963] [ 2162] 0 2162 1049 0 7 26 0 ntpdate
kernel: [ 57.165964] [ 2164] 0 2164 1538 90 9 22 0 lockfile-create
kernel: [ 57.165965] [ 2211] 0 2211 1049 0 8 26 0 ntpdate
kernel: [ 57.165967] [ 2213] 0 2213 1538 82 8 21 0 lockfile-create
kernel: [ 57.165968] [ 2260] 0 2260 1049 0 7 27 0 ntpdate
kernel: [ 57.165970] [ 2262] 0 2262 1538 93 9 22 0 lockfile-create
kernel: [ 57.165971] [ 2309] 0 2309 1049 0 8 26 0 ntpdate
kernel: [ 57.165972] [ 2311] 0 2311 1538 89 9 21 0 lockfile-create
kernel: [ 57.165974] [ 2358] 0 2358 1049 0 8 27 0 ntpdate
kernel: [ 57.165975] [ 2360] 0 2360 1538 94 9 22 0 lockfile-create
kernel: [ 57.165977] [ 2407] 0 2407 1049 0 8 27 0 ntpdate
kernel: [ 57.165978] [ 2409] 0 2409 1538 87 8 21 0 lockfile-create
kernel: [ 57.165979] [ 2456] 0 2456 1049 0 7 26 0 ntpdate
kernel: [ 57.165981] [ 2458] 0 2458 1538 94 8 21 0 lockfile-create
kernel: [ 57.165982] [ 2505] 0 2505 1049 0 7 26 0 ntpdate
kernel: [ 57.165984] [ 2507] 0 2507 1538 94 9 21 0 lockfile-create
kernel: [ 57.165985] [ 2562] 0 2562 1049 0 8 27 0 ntpdate
kernel: [ 57.165986] [ 2564] 0 2564 1538 90 9 22 0 lockfile-create
kernel: [ 57.165988] [ 2621] 0 2621 1049 0 8 26 0 ntpdate
kernel: [ 57.165989] [ 2623] 0 2623 1538 89 8 22 0 lockfile-create
kernel: [ 57.165991] [ 2680] 0 2680 1049 0 8 26 0 ntpdate
kernel: [ 57.165992] [ 2682] 0 2682 1538 80 9 21 0 lockfile-create
kernel: [ 57.165994] [ 2739] 0 2739 1049 0 7 27 0 ntpdate
kernel: [ 57.165995] [ 2741] 0 2741 1538 94 9 21 0 lockfile-create
kernel: [ 57.165996] [ 2798] 0 2798 1049 0 8 26 0 ntpdate
kernel: [ 57.165998] [ 2800] 0 2800 1538 101 9 21 0 lockfile-create
kernel: [ 57.165999] [ 2857] 0 2857 1049 0 8 26 0 ntpdate
kernel: [ 57.166001] [ 2859] 0 2859 1538 90 9 22 0 lockfile-create
kernel: [ 57.166002] [ 2916] 0 2916 1049 0 8 27 0 ntpdate
kernel: [ 57.166003] [ 2918] 0 2918 1538 92 9 22 0 lockfile-create
kernel: [ 57.166005] [ 2975] 0 2975 1049 0 8 27 0 ntpdate
kernel: [ 57.166006] [ 2977] 0 2977 1538 94 9 21 0 lockfile-create
kernel: [ 57.166008] [ 3034] 0 3034 1049 0 8 27 0 ntpdate
kernel: [ 57.166009] [ 3036] 0 3036 1538 93 8 22 0 lockfile-create
kernel: [ 57.166011] [ 3093] 0 3093 1049 0 8 28 0 ntpdate
kernel: [ 57.166012] [ 3095] 0 3095 1538 91 9 22 0 lockfile-create
kernel: [ 57.166013] [ 3152] 0 3152 1049 0 7 27 0 ntpdate
kernel: [ 57.166015] [ 3154] 0 3154 1538 94 9 23 0 lockfile-create
kernel: [ 57.166016] [ 3211] 0 3211 1049 0 7 27 0 ntpdate
kernel: [ 57.166018] [ 3213] 0 3213 1538 102 9 23 0 lockfile-create
kernel: [ 57.166019] [ 3258] 0 3258 1049 0 8 28 0 ntpdate
kernel: [ 57.166020] [ 3260] 0 3260 1538 89 9 22 0 lockfile-create
kernel: [ 57.166022] [ 3317] 0 3317 1049 0 8 28 0 ntpdate
kernel: [ 57.166023] [ 3319] 0 3319 1538 93 8 23 0 lockfile-create
kernel: [ 57.166025] [ 3376] 0 3376 1049 0 8 28 0 ntpdate
kernel: [ 57.166026] [ 3378] 0 3378 1538 80 9 23 0 lockfile-create
kernel: [ 57.166027] [ 3435] 0 3435 1049 0 8 28 0 ntpdate
kernel: [ 57.166029] [ 3437] 0 3437 1538 82 9 22 0 lockfile-create
kernel: [ 57.166030] [ 3494] 0 3494 1049 0 8 28 0 ntpdate
kernel: [ 57.166031] [ 3496] 0 3496 1538 80 9 23 0 lockfile-create
kernel: [ 57.166033] [ 3553] 0 3553 1049 0 8 27 0 ntpdate
kernel: [ 57.166034] [ 3555] 0 3555 1538 90 9 23 0 lockfile-create
kernel: [ 57.166036] [ 3612] 0 3612 1049 0 7 28 0 ntpdate
kernel: [ 57.166037] [ 3614] 0 3614 1538 94 9 23 0 lockfile-create
kernel: [ 57.166038] [ 3671] 0 3671 1049 0 7 28 0 ntpdate
kernel: [ 57.166040] [ 3673] 0 3673 1538 83 9 22 0 lockfile-create
kernel: [ 57.166041] [ 3730] 0 3730 1049 0 7 27 0 ntpdate
kernel: [ 57.166043] [ 3732] 0 3732 1538 101 9 22 0 lockfile-create
kernel: [ 57.166044] [ 3789] 0 3789 1049 0 8 27 0 ntpdate
kernel: [ 57.166046] [ 3791] 0 3791 1538 87 9 22 0 lockfile-create
kernel: [ 57.166047] [ 3848] 0 3848 1049 0 8 28 0 ntpdate
kernel: [ 57.166049] [ 3850] 0 3850 1538 92 8 23 0 lockfile-create
kernel: [ 57.166050] [ 3907] 0 3907 1049 0 7 28 0 ntpdate
kernel: [ 57.166051] [ 3909] 0 3909 1538 92 9 22 0 lockfile-create
kernel: [ 57.166053] [ 3966] 0 3966 1049 0 7 28 0 ntpdate
kernel: [ 57.166054] [ 3968] 0 3968 1538 93 9 23 0 lockfile-create
kernel: [ 57.166056] [ 4025] 0 4025 1049 0 8 28 0 ntpdate
kernel: [ 57.166057] [ 4027] 0 4027 1538 88 9 22 0 lockfile-create
kernel: [ 57.166059] [ 4084] 0 4084 1049 0 8 27 0 ntpdate
kernel: [ 57.166060] [ 4086] 0 4086 1538 87 8 23 0 lockfile-create
kernel: [ 57.166062] [ 4143] 0 4143 1049 0 8 28 0 ntpdate
kernel: [ 57.166063] [ 4145] 0 4145 1538 102 9 23 0 lockfile-create
kernel: [ 57.166064] [ 4202] 0 4202 1049 0 8 28 0 ntpdate
kernel: [ 57.166066] [ 4204] 0 4204 1538 91 9 23 0 lockfile-create
kernel: [ 57.166067] [ 4261] 0 4261 1049 0 8 27 0 ntpdate
kernel: [ 57.166069] [ 4263] 0 4263 1538 84 7 22 0 lockfile-create
kernel: [ 57.166070] [ 4320] 0 4320 1049 0 8 27 0 ntpdate
kernel: [ 57.166072] [ 4322] 0 4322 1538 88 9 22 0 lockfile-create
kernel: [ 57.166073] [ 4379] 0 4379 1049 0 8 27 0 ntpdate
kernel: [ 57.166074] [ 4381] 0 4381 1538 82 9 22 0 lockfile-create
kernel: [ 57.166076] [ 4438] 0 4438 1049 0 8 27 0 ntpdate
kernel: [ 57.166077] [ 4440] 0 4440 1538 82 9 23 0 lockfile-create
kernel: [ 57.166079] [ 4497] 0 4497 1049 0 8 27 0 ntpdate
kernel: [ 57.166080] [ 4499] 0 4499 1538 101 8 23 0 lockfile-create
kernel: [ 57.166082] [ 4556] 0 4556 1049 0 8 27 0 ntpdate
kernel: [ 57.166083] [ 4558] 0 4558 1538 84 9 23 0 lockfile-create
kernel: [ 57.166085] [ 4615] 0 4615 1049 0 8 27 0 ntpdate
kernel: [ 57.166086] [ 4617] 0 4617 1538 91 8 22 0 lockfile-create
kernel: [ 57.166087] [ 4674] 0 4674 1049 0 8 27 0 ntpdate
kernel: [ 57.166089] [ 4676] 0 4676 1538 91 9 23 0 lockfile-create
kernel: [ 57.166090] [ 4733] 0 4733 1049 0 7 27 0 ntpdate
kernel: [ 57.166092] [ 4735] 0 4735 1538 104 9 17 0 lockfile-create
kernel: [ 57.166093] [ 4792] 0 4792 1049 0 9 27 0 ntpdate
kernel: [ 57.166094] [ 4794] 0 4794 1538 93 9 16 0 lockfile-create
kernel: [ 57.166096] [ 4851] 0 4851 1049 0 7 28 0 ntpdate
kernel: [ 57.166097] [ 4853] 0 4853 1538 84 9 23 0 lockfile-create
kernel: [ 57.166098] [ 4919] 0 4919 1049 0 8 28 0 ntpdate
kernel: [ 57.166100] [ 4921] 0 4921 1538 94 8 22 0 lockfile-create
kernel: [ 57.166101] [ 4975] 0 4975 1049 0 8 28 0 ntpdate
kernel: [ 57.166103] [ 4977] 0 4977 1538 101 9 22 0 lockfile-create
kernel: [ 57.166104] [ 5031] 0 5031 1049 0 7 28 0 ntpdate
kernel: [ 57.166105] [ 5033] 0 5033 1538 91 8 23 0 lockfile-create
kernel: [ 57.166107] [ 5087] 0 5087 1049 0 8 27 0 ntpdate
kernel: [ 57.166108] [ 5089] 0 5089 1538 101 8 23 0 lockfile-create
kernel: [ 57.166110] [ 5123] 0 5123 1049 0 7 27 0 ntpdate
kernel: [ 57.166111] [ 5125] 0 5125 1538 102 9 22 0 lockfile-create
kernel: [ 57.166112] [ 5142] 0 5142 1028 282 8 26 0 startpar
kernel: [ 57.166114] [ 5155] 0 5155 4780 459 15 65 0 rpcbind
kernel: [ 57.166115] [ 6387] 0 6387 1049 337 8 40 0 rc
kernel: [ 57.166117] [ 6396] 0 6396 3261 707 11 0 0 startpar
kernel: [ 57.166118] [ 6559] 0 6559 32047 255 25 302 0 rsyslogd
kernel: [ 57.166120] [ 6617] 0 6617 4568 189 13 111 0 starter
kernel: [ 57.166121] [ 6619] 0 6619 78386 1012 81 420 0 charon
kernel: [ 57.166122] [ 6688] 0 6688 3289 365 11 56 0 mdadm
kernel: [ 57.166124] [ 6722] 0 6722 4172 0 12 39 0 atd
kernel: [ 57.166125] [ 6769] 0 6769 1033 354 7 36 0 acpid
kernel: [ 57.166127] [ 6833] 0 6833 5142 444 14 71 0 irqbalance
kernel: [ 57.166128] [ 6839] 0 6839 11807 577 26 85 0 ha_logd
kernel: [ 57.166130] [ 6861] 108 6861 6425 0 17 60 0 dbus-daemon
kernel: [ 57.166131] [ 6879] 109 6879 9180 243 19 335 0 apt-cacher-ng
kernel: [ 57.166132] [ 6902] 0 6902 11807 455 25 84 0 ha_logd
kernel: [ 57.166134] [ 6927] 0 6927 3487 519 10 198 0 conntrackd
kernel: [ 57.166135] [ 6944] 0 6944 15452 521 29 259 0 bacula-fd
kernel: [ 57.166136] [ 6968] 0 6968 5232 427 16 62 0 cron
kernel: [ 57.166138] [ 7173] 0 7173 56058 1226 70 628 0 corosync
kernel: [ 57.166139] [ 7253] 106 7253 24135 1625 52 1850 0 cib
kernel: [ 57.166141] [ 7254] 0 7254 22205 1242 46 152 0 stonithd
kernel: [ 57.166142] [ 7255] 0 7255 26022 1286 52 254 0 lrmd
kernel: [ 57.166144] [ 7256] 106 7256 23284 1026 49 138 0 attrd
kernel: [ 57.166145] [ 7257] 106 7257 22318 1028 47 136 0 pengine
kernel: [ 57.166146] [ 7258] 106 7258 23861 1310 51 686 0 crmd
kernel: [ 57.166148] [ 7259] 0 7259 3397 20 9 1365 0 dhcpd
kernel: [ 57.166149] [ 7415] 101 7415 12285 596 25 125 0 exim4
kernel: [ 57.166151] [ 7553] 1003 7553 5814 452 16 126 0 nrpe
kernel: [ 57.166152] [ 7640] 111 7640 9745 754 24 106 0 ntpd
kernel: [ 57.166154] [ 7701] 0 7701 2335 633 9 34 0 providers_super
kernel: [ 57.166155] [ 7797] 0 7797 1563 377 9 28 0 lsm
kernel: [ 57.166156] [ 7809] 115 7809 6579 466 17 142 0 zebra
kernel: [ 57.166158] [ 7836] 115 7836 7315 702 17 197 0 ospfd
kernel: [ 57.166159] [ 7849] 0 7849 4294 377 13 50 0 watchquagga
kernel: [ 57.166160] [ 7901] 0 7901 2315 600 10 22 0 routewatch_supe
kernel: [ 57.166162] [ 7943] 0 7943 2378 607 11 72 0 routeupdate_sup
kernel: [ 57.166163] [ 8813] 0 8813 1538 96 9 22 0 lockfile-touch
kernel: [ 57.166164] [ 8814] 0 8814 5756 512 17 94 0 ntpdate
kernel: [ 57.166166] [11698] 0 11698 12364 1182 30 191 0 exim4
kernel: [ 57.166167] [12220] 0 12220 5504 242 14 85 0 keepalived
kernel: [ 57.166169] [12222] 0 12222 7639 766 20 134 0 keepalived
kernel: [ 57.166170] [12223] 0 12223 7607 437 19 110 0 keepalived
kernel: [ 57.166172] [12319] 0 12319 12285 1026 28 56 0 exim4
kernel: [ 57.166173] [12381] 0 12381 10073 370 23 216 0 udevd
kernel: [ 57.166174] [12382] 0 12382 10073 259 23 236 0 udevd
kernel: [ 57.166176] [12593] 0 12593 1049 0 8 28 0 ntpdate
kernel: [ 57.166177] [12595] 0 12595 1538 94 9 23 0 lockfile-create
kernel: [ 57.166179] [12779] 0 12779 1049 0 8 27 0 ntpdate
kernel: [ 57.166180] [12781] 0 12781 1538 95 9 23 0 lockfile-create
kernel: [ 57.166182] [12810] 0 12810 13069 516 28 158 -1000 sshd
kernel: [ 57.166183] [12963] 0 12963 1049 349 8 13 0 supervisor_scri
kernel: [ 57.166185] [12981] 0 12981 1049 327 8 38 0 rc.local
kernel: [ 57.166186] [13003] 0 13003 1049 103 7 22 0 rc.local
kernel: [ 57.166187] [13004] 0 13004 1190 351 8 178 0 shorewall
kernel: [ 57.166189] [13046] 0 13046 3663 331 14 2636 0 sh
kernel: [ 57.166190] [14984] 0 14984 1082 0 7 66 0 VIPaddr
kernel: [ 57.166192] [15056] 0 15056 34508 20983 73 9410 0 iptables
kernel: [ 57.166193] [15121] 0 15121 1082 0 7 66 0 VIPaddr
kernel: [ 57.166194] [15259] 0 15259 1082 54 7 13 0 VIPaddr
kernel: [ 57.166196] [15260] 0 15260 9720 280 23 47 0 send_arp
kernel: [ 57.166197] [15374] 0 15374 1082 52 8 15 0 VIPaddr
kernel: [ 57.166199] [15375] 0 15375 9720 316 23 0 0 send_arp
kernel: [ 57.166200] [15430] 0 15430 1082 409 7 0 0 VIPaddr
kernel: [ 57.166201] [15431] 0 15431 1082 415 7 0 0 VIPaddr
kernel: [ 57.166203] [15467] 0 15467 1049 271 8 6 0 supervisor_scri
kernel: [ 57.166204] [15471] 0 15471 1028 114 8 0 0 sleep
kernel: [ 57.166206] [15478] 0 15478 1049 114 7 0 0 lsmd
kernel: [ 57.166207] [15482] 0 15482 1049 188 8 6 0 supervisor_scri
kernel: [ 57.166208] [15483] 0 15483 3866 530 13 0 0 ps
kernel: [ 57.166210] [15484] 0 15484 1580 118 9 0 0 grep
kernel: [ 57.166211] [15485] 0 15485 1580 120 9 0 0 grep
kernel: [ 57.166212] [15486] 0 15486 1580 109 8 0 0 grep
kernel: [ 57.166214] [15487] 0 15487 1580 106 9 0 0 grep
kernel: [ 57.166215] [15490] 0 15490 1580 118 9 0 0 grep
kernel: [ 57.166216] [15493] 0 15493 1033 132 8 0 0 cat
kernel: [ 57.166218] [15494] 0 15494 2315 402 10 20 0 routewatch_supe
kernel: [ 57.166219] [15499] 0 15499 2315 35 10 20 0 routewatch_supe
kernel: [ 57.166221] [15500] 0 15500 1082 65 7 0 0 VIPaddr
kernel: [ 57.166222] [15501] 0 15501 2335 412 9 30 0 providers_super
kernel: [ 57.166223] [15503] 0 15503 2335 45 9 30 0 providers_super
kernel: [ 57.166225] [15504] 0 15504 2335 45 9 30 0 providers_super
kernel: [ 57.166226] [15506] 0 15506 1082 64 7 0 0 VIPaddr
kernel: [ 57.166228] [15507] 0 15507 26022 558 50 242 0 lrmd
kernel: [ 57.166229] Out of memory: Kill process 15056 (iptables) score 8 or sacrifice child
kernel: [ 57.166457] Killed process 15056 (iptables) total-vm:138032kB, anon-rss:82552kB, file-rss:1380kB
