My fresh new Ubuntu webserver randomly crashes with the following error in the syslog. It doesn't respond anymore and I have to do a hard reboot. It happens almost one time a day.
Server kernel is : Linux 3.10.23-xxxx-std-ipv6-64 #1 SMP Mon Dec 9 18:53:52 CET 2013 x86_64 x86_64 x86_64 GNU/Linux
I tried memtester but I didn't get any error.
Any idea ?
Mar 17 12:58:25 hephaistos kernel: BUG: unable to handle kernel paging request at ffff880562ea7f00
Mar 17 12:58:25 hephaistos kernel: IP: [<ffffffff811612e0>] kmem_cache_alloc+0x50/0xd0
Mar 17 12:58:25 hephaistos kernel: PGD 23e1067 PUD 0
Mar 17 12:58:25 hephaistos kernel: Oops: 0000 [#1] SMP
Mar 17 12:58:25 hephaistos kernel: CPU: 4 PID: 12525 Comm: apache2 Not tainted 3.10.23-xxxx-std-ipv6-64 #1
Mar 17 12:58:25 hephaistos kernel: Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Mar 17 12:58:25 hephaistos kernel: task: ffff880468e61a80 ti: ffff88026bc86000 task.ti: ffff88026bc86000
Mar 17 12:58:25 hephaistos kernel: RIP: 0010:[<ffffffff811612e0>] [<ffffffff811612e0>] kmem_cache_alloc+0x50/0xd0
Mar 17 12:58:25 hephaistos kernel: RSP: 0018:ffff88026bc87db0 EFLAGS: 00010286
Mar 17 12:58:25 hephaistos kernel: RAX: 0000000000000000 RBX: ffff880468e61a80 RCX: 0000000000558cea
Mar 17 12:58:25 hephaistos kernel: RDX: 0000000000558ce9 RSI: 00000000000000d0 RDI: ffffffff810e1b73
Mar 17 12:58:25 hephaistos kernel: RBP: ffff88026bc87dd0 R08: 0000000000012a00 R09: 0000000000000000
Mar 17 12:58:25 hephaistos kernel: R10: 0000000000000000 R11: ffffffffffffffc0 R12: ffff880562ea7f00
Mar 17 12:58:25 hephaistos kernel: R13: ffff88046f003800 R14: 00000000000000d0 R15: ffff880319132e60
Mar 17 12:58:25 hephaistos kernel: FS: 00007f165c437780(0000) GS:ffff88047fd00000(0000) knlGS:0000000000000000
Mar 17 12:58:25 hephaistos kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 17 12:58:25 hephaistos kernel: CR2: ffff880562ea7f00 CR3: 0000000462d33000 CR4: 00000000001007e0
Mar 17 12:58:25 hephaistos kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 17 12:58:25 hephaistos kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 17 12:58:25 hephaistos kernel: Stack:
Mar 17 12:58:25 hephaistos kernel: ffff880468e61a80 ffff880319132e60 0000000001200011 ffff8802166f6000
Mar 17 12:58:25 hephaistos kernel: ffff88026bc87df0 ffffffff810e1b73 ffff88047ffeed00 0000000001200011
Mar 17 12:58:25 hephaistos kernel: ffff88026bc87e30 ffffffff810e1d21 0000000000000000 ffff8802166f6000
Mar 17 12:58:25 hephaistos kernel: Call Trace:
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1d21>] copy_creds+0x31/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810ba82f>] copy_process.part.41+0x24f/0x13d0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b212>] ? __alloc_fd+0x42/0x100
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bba9f>] do_fork+0xaf/0x300
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b2fb>] ? get_unused_fd_flags+0x2b/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81179e37>] ? __do_pipe_flags+0x77/0xb0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b376>] ? __fd_install+0x26/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bbd71>] SyS_clone+0x11/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d324b9>] stub_clone+0x69/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d32212>] ? system_call_fastpath+0x16/0x1b
Mar 17 12:58:25 hephaistos kernel: Code: 00 65 4c 03 04 25 a8 cc 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 74 51 48 85 c0 74 4c 49 63 45 20 4d 8b 45 00 48 8d 4a 01 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bf 49 63
Mar 17 12:58:25 hephaistos kernel: RIP [<ffffffff811612e0>] kmem_cache_alloc+0x50/0xd0
Mar 17 12:58:25 hephaistos kernel: RSP <ffff88026bc87db0>
Mar 17 12:58:25 hephaistos kernel: CR2: ffff880562ea7f00
Mar 17 12:58:25 hephaistos kernel: ---[ end trace 894883cad0d802a3 ]---
Mar 17 12:58:25 hephaistos kernel: ------------[ cut here ]------------
Mar 17 12:58:25 hephaistos kernel: WARNING: at kernel/smp.c:382 smp_call_function_many+0xaf/0x2d0()
Mar 17 12:58:25 hephaistos kernel: CPU: 4 PID: 12525 Comm: apache2 Tainted: G D 3.10.23-xxxx-std-ipv6-64 #1
Mar 17 12:58:25 hephaistos kernel: Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Mar 17 12:58:25 hephaistos kernel: 0000000000000009 ffff88026bc87808 ffffffff81d293c4 ffff88026bc87848
Mar 17 12:58:25 hephaistos kernel: ffffffff810bc696 ffff88026bc879a8 ffff88045cbb4980 ffff88045cbb4c48
Mar 17 12:58:25 hephaistos kernel: 0000000000000004 0000000000000000 ffff880468e61a80 ffff88026bc87858
Mar 17 12:58:25 hephaistos kernel: Call Trace:
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d293c4>] dump_stack+0x19/0x1b
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bc696>] warn_slowpath_common+0x66/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bc795>] warn_slowpath_null+0x15/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810fed9f>] smp_call_function_many+0xaf/0x2d0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070bb0>] ? do_flush_tlb_all+0x60/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070c79>] native_flush_tlb_others+0x29/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070ee4>] flush_tlb_mm_range+0x1f4/0x230
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113fe47>] tlb_flush_mmu+0x37/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113feaf>] tlb_finish_mmu+0xf/0x40
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811498bc>] exit_mmap+0xec/0x170
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810b9cf3>] mmput+0x43/0xd0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810c1794>] do_exit+0x254/0x990
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bf48c>] ? kmsg_dump+0x9c/0xc0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81043cb6>] oops_end+0x96/0xe0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d21f78>] no_context+0x24e/0x279
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d22158>] __bad_area_nosemaphore+0x1b5/0x1d4
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d22185>] bad_area_nosemaphore+0xe/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8106c4d2>] __do_page_fault+0x292/0x4b0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113a829>] ? zone_statistics+0x99/0xc0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811233c2>] ? unlock_page+0x22/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8106c719>] do_page_fault+0x9/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d31cf2>] page_fault+0x22/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] ? prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811612e0>] ? kmem_cache_alloc+0x50/0xd0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1d21>] copy_creds+0x31/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810ba82f>] copy_process.part.41+0x24f/0x13d0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b212>] ? __alloc_fd+0x42/0x100
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bba9f>] do_fork+0xaf/0x300
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b2fb>] ? get_unused_fd_flags+0x2b/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81179e37>] ? __do_pipe_flags+0x77/0xb0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b376>] ? __fd_install+0x26/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bbd71>] SyS_clone+0x11/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d324b9>] stub_clone+0x69/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d32212>] ? system_call_fastpath+0x16/0x1b
Mar 17 12:58:25 hephaistos kernel: ---[ end trace 894883cad0d802a4 ]---
Mar 17 12:58:25 hephaistos kernel: ------------[ cut here ]------------
Mar 17 12:58:25 hephaistos kernel: WARNING: at kernel/smp.c:244 smp_call_function_single+0x11b/0x160()
Mar 17 12:58:25 hephaistos kernel: CPU: 4 PID: 12525 Comm: apache2 Tainted: G D W 3.10.23-xxxx-std-ipv6-64 #1
Mar 17 12:58:25 hephaistos kernel: Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Mar 17 12:58:25 hephaistos kernel: 0000000000000009 ffff88026bc87798 ffffffff81d293c4 ffff88026bc877d8
Mar 17 12:58:25 hephaistos kernel: ffffffff810bc696 ffffffff81fea74d 0000000000000002 0000000000000004
Mar 17 12:58:25 hephaistos kernel: ffffffff821dba00 ffffffff81070bb0 ffff880468e61a80 ffff88026bc877e8
Mar 17 12:58:25 hephaistos kernel: Call Trace:
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d293c4>] dump_stack+0x19/0x1b
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bc696>] warn_slowpath_common+0x66/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070bb0>] ? do_flush_tlb_all+0x60/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bc795>] warn_slowpath_null+0x15/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810feaeb>] smp_call_function_single+0x11b/0x160
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810fef72>] smp_call_function_many+0x282/0x2d0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070bb0>] ? do_flush_tlb_all+0x60/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070c79>] native_flush_tlb_others+0x29/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81070ee4>] flush_tlb_mm_range+0x1f4/0x230
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113fe47>] tlb_flush_mmu+0x37/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113feaf>] tlb_finish_mmu+0xf/0x40
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811498bc>] exit_mmap+0xec/0x170
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810b9cf3>] mmput+0x43/0xd0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810c1794>] do_exit+0x254/0x990
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bf48c>] ? kmsg_dump+0x9c/0xc0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81043cb6>] oops_end+0x96/0xe0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d21f78>] no_context+0x24e/0x279
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d22158>] __bad_area_nosemaphore+0x1b5/0x1d4
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d22185>] bad_area_nosemaphore+0xe/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8106c4d2>] __do_page_fault+0x292/0x4b0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113a829>] ? zone_statistics+0x99/0xc0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811233c2>] ? unlock_page+0x22/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8106c719>] do_page_fault+0x9/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d31cf2>] page_fault+0x22/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] ? prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811612e0>] ? kmem_cache_alloc+0x50/0xd0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1d21>] copy_creds+0x31/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810ba82f>] copy_process.part.41+0x24f/0x13d0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b212>] ? __alloc_fd+0x42/0x100
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bba9f>] do_fork+0xaf/0x300
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b2fb>] ? get_unused_fd_flags+0x2b/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81179e37>] ? __do_pipe_flags+0x77/0xb0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b376>] ? __fd_install+0x26/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bbd71>] SyS_clone+0x11/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d324b9>] stub_clone+0x69/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d32212>] ? system_call_fastpath+0x16/0x1b
Mar 17 12:58:25 hephaistos kernel: ---[ end trace 894883cad0d802a5 ]---
Mar 17 12:58:25 hephaistos kernel: ------------[ cut here ]------------
Mar 17 12:58:25 hephaistos kernel: WARNING: at kernel/softirq.c:160 local_bh_enable+0x72/0xa0()
Mar 17 12:58:25 hephaistos kernel: CPU: 4 PID: 12525 Comm: apache2 Tainted: G D W 3.10.23-xxxx-std-ipv6-64 #1
Mar 17 12:58:25 hephaistos kernel: Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0b 11/06/2013
Mar 17 12:58:25 hephaistos kernel: 0000000000000009 ffff88026bc878a8 ffffffff81d293c4 ffff88026bc878e8
Mar 17 12:58:25 hephaistos kernel: ffffffff810bc696 ffff8803543a4500 ffff88045a42e180 ffff880004bfd180
Mar 17 12:58:25 hephaistos kernel: ffff8803ee978f00 0000000000000000 ffff8804694a6620 ffff88026bc878f8
Mar 17 12:58:25 hephaistos kernel: Call Trace:
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d293c4>] dump_stack+0x19/0x1b
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bc696>] warn_slowpath_common+0x66/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bc795>] warn_slowpath_null+0x15/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810c3772>] local_bh_enable+0x72/0xa0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81b42726>] lock_sock_nested+0x36/0x40
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81bdbb98>] tcp_close+0x18/0x410
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81c001bf>] inet_release+0x6f/0x80
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81c40fa4>] inet6_release+0x34/0x50
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81b3dc18>] sock_release+0x18/0x80
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81b3de9d>] sock_close+0xd/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811727d1>] __fput+0xe1/0x220
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81172919>] ____fput+0x9/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810d9585>] task_work_run+0xb5/0xd0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810c17c1>] do_exit+0x281/0x990
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bf48c>] ? kmsg_dump+0x9c/0xc0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81043cb6>] oops_end+0x96/0xe0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d21f78>] no_context+0x24e/0x279
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d22158>] __bad_area_nosemaphore+0x1b5/0x1d4
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d22185>] bad_area_nosemaphore+0xe/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8106c4d2>] __do_page_fault+0x292/0x4b0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8113a829>] ? zone_statistics+0x99/0xc0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811233c2>] ? unlock_page+0x22/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8106c719>] do_page_fault+0x9/0x10
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d31cf2>] page_fault+0x22/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] ? prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff811612e0>] ? kmem_cache_alloc+0x50/0xd0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1b73>] prepare_creds+0x23/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810e1d21>] copy_creds+0x31/0x150
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810ba82f>] copy_process.part.41+0x24f/0x13d0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b212>] ? __alloc_fd+0x42/0x100
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bba9f>] do_fork+0xaf/0x300
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b2fb>] ? get_unused_fd_flags+0x2b/0x30
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81179e37>] ? __do_pipe_flags+0x77/0xb0
Mar 17 12:58:25 hephaistos kernel: [<ffffffff8118b376>] ? __fd_install+0x26/0x60
Mar 17 12:58:25 hephaistos kernel: [<ffffffff810bbd71>] SyS_clone+0x11/0x20
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d324b9>] stub_clone+0x69/0x90
Mar 17 12:58:25 hephaistos kernel: [<ffffffff81d32212>] ? system_call_fastpath+0x16/0x1b
Mar 17 12:58:25 hephaistos kernel: ---[ end trace 894883cad0d802a6 ]---
