I have Dell PowerEdge 1950 with debian which reboot without reason, and after reboot BIOS write that "previous shutdown was with error press f1 for continue".
First of all, is it possible switch of this "f1 for continue"? Its remote server and I hate this.
And I can not explain - why its rebooted. There is part of my logs:
Sep 7 11:48:11 kernel: [182893.472177] virbr0: port 8(vnet7) entered forwarding state
Sep 7 11:48:11 kernel: [182893.484451] libvirtd: page allocation failure: order:4, mode:0x2000d0
Sep 7 11:48:11 kernel: [182893.484456] CPU: 0 PID: 2427 Comm: libvirtd Not tainted 3.10-2-amd64 #1 Debian 3.10.7-1
Sep 7 11:48:11 kernel: [182893.484458] Hardware name: Dell Inc. PowerEdge 1950/0DT097, BIOS 2.7.0 10/30/2010
Sep 7 11:48:11 kernel: [182893.484461] 0000000000000000 ffffffff810c4af9 ffff88012fff9b08 ffff88012fff8d80
Sep 7 11:48:11 kernel: [182893.484466] 0000000000000000 0000000000000000 ffff880100000002 ffffffff810d4f89
Sep 7 11:48:11 kernel: [182893.484469] 0000000000000010 fffffffffffffff0 000000000000001e 0000000000000000
Sep 7 11:48:11 kernel: [182893.484473] Call Trace:
Sep 7 11:48:11 kernel: [182893.484482] [<ffffffff810c4af9>] ? warn_alloc_failed+0x118/0x12c
Sep 7 11:48:11 kernel: [182893.484487] [<ffffffff810d4f89>] ? next_online_pgdat+0x1c/0x3a
Sep 7 11:48:11 kernel: [182893.484491] [<ffffffff810c7469>] ? __alloc_pages_nodemask+0x206/0x78e
Sep 7 11:48:11 kernel: [182893.484496] [<ffffffff810f90dc>] ? kmem_getpages+0x53/0x122
Sep 7 11:48:11 kernel: [182893.484499] [<ffffffff810f997c>] ? fallback_alloc+0x123/0x1e5
Sep 7 11:48:11 kernel: [182893.484502] [<ffffffff810f9efa>] ? kmem_cache_alloc_trace+0x7c/0xcc
Sep 7 11:48:11 kernel: [182893.484510] [<ffffffffa0543fcc>] ? vhost_net_open+0x1f/0x17d [vhost_net]
Sep 7 11:48:11 kernel: [182893.484515] [<ffffffff8126a54e>] ? misc_open+0x117/0x17a
Sep 7 11:48:11 kernel: [182893.484519] [<ffffffff8110cd6c>] ? chrdev_open+0x11c/0x140
Sep 7 11:48:11 kernel: [182893.484522] [<ffffffff8110cc50>] ? cdev_put+0x19/0x19
Sep 7 11:48:11 kernel: [182893.484526] [<ffffffff8110815b>] ? do_dentry_open+0x16d/0x20f
Sep 7 11:48:11 kernel: [182893.484529] [<ffffffff81108227>] ? finish_open+0x2a/0x33
Sep 7 11:48:11 kernel: [182893.484533] [<ffffffff811146d7>] ? do_last+0x864/0xa0c
Sep 7 11:48:11 kernel: [182893.484536] [<ffffffff8111219f>] ? __inode_permission+0x57/0x95
Sep 7 11:48:11 kernel: [182893.484539] [<ffffffff81114941>] ? path_openat+0xc2/0x327
Sep 7 11:48:11 kernel: [182893.484543] [<ffffffff81114e49>] ? do_filp_open+0x2a/0x6e
Sep 7 11:48:11 kernel: [182893.484546] [<ffffffff810fa414>] ? kmem_cache_alloc+0x8c/0xfd
Sep 7 11:48:11 kernel: [182893.484550] [<ffffffff8111dcce>] ? __alloc_fd+0xcc/0xdb
Sep 7 11:48:11 kernel: [182893.484553] [<ffffffff81108a99>] ? do_sys_open+0x5c/0xe0
Sep 7 11:48:11 kernel: [182893.484557] [<ffffffff8138ade9>] ? system_call_fastpath+0x16/0x1b
Sep 7 11:48:11 kernel: [182893.484559] Mem-Info:
Sep 7 11:48:11 kernel: [182893.484561] Node 0 DMA per-cpu:
Sep 7 11:48:11 kernel: [182893.484564] CPU 0: hi: 0, btch: 1 usd: 0
Sep 7 11:48:11 kernel: [182893.484566] CPU 1: hi: 0, btch: 1 usd: 0
Sep 7 11:48:11 kernel: [182893.484568] CPU 2: hi: 0, btch: 1 usd: 0
Sep 7 11:48:11 kernel: [182893.484569] CPU 3: hi: 0, btch: 1 usd: 0
Sep 7 11:48:11 kernel: [182893.484571] Node 0 DMA32 per-cpu:
Sep 7 11:48:11 kernel: [182893.484573] CPU 0: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484575] CPU 1: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484577] CPU 2: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484579] CPU 3: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484581] Node 0 Normal per-cpu:
Sep 7 11:48:11 kernel: [182893.484583] CPU 0: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484585] CPU 1: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484587] CPU 2: hi: 186, btch: 31 usd: 2
Sep 7 11:48:11 kernel: [182893.484589] CPU 3: hi: 186, btch: 31 usd: 0
Sep 7 11:48:11 kernel: [182893.484594] active_anon:423230 inactive_anon:137000 isolated_anon:0
Sep 7 11:48:11 kernel: [182893.484594] active_file:169377 inactive_file:162749 isolated_file:0
Sep 7 11:48:11 kernel: [182893.484594] unevictable:0 dirty:7 writeback:0 unstable:0
Sep 7 11:48:11 kernel: [182893.484594] free:57651 slab_reclaimable:16563 slab_unreclaimable:9296
Sep 7 11:48:11 kernel: [182893.484594] mapped:2985 shmem:145 pagetables:3918 bounce:0
Sep 7 11:48:11 kernel: [182893.484594] free_cma:0
Sep 7 11:48:11 kernel: [182893.484599] Node 0 DMA free:15908kB min:264kB low:328kB high:396kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Sep 7 11:48:11 kernel: [182893.484607] lowmem_reserve[]: 0 3231 3940 3940
Sep 7 11:48:11 kernel: [182893.484610] Node 0 DMA32 free:188268kB min:55212kB low:69012kB high:82816kB active_anon:1429476kB inactive_anon:284516kB active_file:658168kB inactive_file:631332kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3386688kB managed:3309356kB mlocked:0kB dirty:16kB writeback:0kB mapped:6836kB shmem:220kB slab_reclaimable:53080kB slab_unreclaimable:12744kB kernel_stack:1024kB pagetables:7692kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep 7 11:48:11 kernel: [182893.484618] lowmem_reserve[]: 0 0 708 708
Sep 7 11:48:11 kernel: [182893.484621] Node 0 Normal free:26428kB min:12100kB low:15124kB high:18148kB active_anon:263444kB inactive_anon:263484kB active_file:19340kB inactive_file:19664kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:786432kB managed:725380kB mlocked:0kB dirty:12kB writeback:0kB mapped:5104kB shmem:360kB slab_reclaimable:13172kB slab_unreclaimable:24440kB kernel_stack:1400kB pagetables:7980kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep 7 11:48:11 kernel: [182893.484629] lowmem_reserve[]: 0 0 0 0
Sep 7 11:48:11 kernel: [182893.484632] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15908kB
Sep 7 11:48:11 kernel: [182893.484645] Node 0 DMA32: 12519*4kB (UEM) 12482*8kB (UEM) 2152*16kB (UM) 3*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 188556kB
Sep 7 11:48:11 kernel: [182893.484659] Node 0 Normal: 5894*4kB (UM) 108*8kB (UM) 1*16kB (R) 2*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 27336kB
Sep 7 11:48:11 kernel: [182893.484684] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 7 11:48:11 kernel: [182893.484685] 345767 total pagecache pages
Sep 7 11:48:11 kernel: [182893.484688] 13606 pages in swap cache
Sep 7 11:48:11 kernel: [182893.484690] Swap cache stats: add 110606, delete 97000, find 86650/91247
Sep 7 11:48:11 kernel: [182893.484692] Free swap = 2779960kB
Sep 7 11:48:11 kernel: [182893.484693] Total swap = 2928636kB
Sep 7 11:48:11 kernel: [182893.504936] 1048575 pages RAM
Sep 7 11:48:11 kernel: [182893.504939] 34669 pages reserved
Sep 7 11:48:11 kernel: [182893.504940] 1385978 pages shared
Sep 7 11:48:11 kernel: [182893.504941] 629752 pages non-shared
Sep 7 11:48:11 kernel: [182893.504944] kmem_getpages: 150006 callbacks suppressed
Sep 7 11:48:11 kernel: [182893.504946] SLAB: Unable to allocate memory on node 0 (gfp=0xd0)
Sep 7 11:48:11 kernel: [182893.504949] cache: kmalloc-65536, object size: 65536, order: 4
Sep 7 11:48:11 kernel: [182893.504953] node 0: slabs: 13/13, objs: 13/13, free: 0
Sep 7 11:52:49 kernel: [183171.325360] kvm: sending ioctl 5326 to a partition!
Sep 7 11:52:49 kernel: [183171.325374] kvm: sending ioctl 80200204 to a partition!
What is faulted? Memory allocation? I have some kvm virtual machines on this host , but its sums of memory less then on hosts, and I have a lot of free swap.
UPD: new log, now with SMP string:
Sep 10 17:03:54 kernel: [83522.369641] Hardware name: Dell Inc. PowerEdge 1950/0DT097, BIOS 2.7.0 10/30/2010
Sep 10 17:03:54 kernel: [83522.369644] 0000000000000000 ffffffffa0352060 0000000000000000 00000001087ef000 Sep 10 17:03:54 kernel: [83522.369649] ffff8800c6440080 0000000000000000 0000000000000000 ffff8800c6440080 Sep 10 17:03:54 kernel: [83522.369652] 0000000000000000 0000000000000001 ffffffffa034b051 0000000100000008 Sep 10 17:03:54 kernel: [83522.369656] Call Trace:
Sep 10 17:03:54 kernel: [83522.369703] [] ? kvm_mmu_load+0x315/0x33b [kvm] Sep 10 17:03:54 kernel: [83522.369716] [] ? kvm_arch_vcpu_ioctl_run+0x8b4/0xe9f [kvm] Sep 10 17:03:54 kernel: [83522.369723] [] ? vmx_vcpu_load+0x28/0x14e [kvm_intel] Sep 10 17:03:54 kernel: [83522.369728] [] ? paravirt_write_msr+0xb/0xe [kvm_intel]
Sep 10 17:03:54 kernel: [83522.369736] [] ? kvm_vcpu_ioctl+0x116/0x48b [kvm] Sep 10 17:03:54 kernel: [83522.369743] [] ? do_futex+0xd1/0x7b5
Sep 10 17:03:54 kernel: [83522.369748] [] ? do_sync_readv_writev+0x76/0x76 Sep 10 17:03:54 kernel: [83522.369752] [] ? vfs_ioctl+0x1b/0x25
Sep 10 17:03:54 kernel: [83522.369755] [] ? do_vfs_ioctl+0x3e8/0x42a Sep 10 17:03:54 kernel: [83522.369759] [] ? finish_task_switch+0x48/0xaa
Sep 10 17:03:54 kernel: [83522.369764] [] ? __schedule+0x4dc/0x532 Sep 10 17:03:54 kernel: [83522.369767] [] ? SyS_ioctl+0x4e/0x79
Sep 10 17:03:54 kernel: [83522.369771] [] ? system_call_fastpath+0x16/0x1b Sep 11 12:10:19 kernel: [152307.503781] BUG: unable to handle kernel NULL pointer dereference at 000000000000021c
Sep 11 12:10:19 kernel: [152307.503946] IP: [] tag_get+0x2/0xd Sep 11 12:10:19 kernel: [152307.504038] PGD 0-
Sep 11 12:10:19 kernel: [152307.504081] Oops: 0000 [#1] SMP-
UPD:
Screenshot from one of debian virtual machine inside kvm:
After that real server was frozen, and rebooted with power