how to turn a rescue kernel into a working one?

Question

A yum update today made one of my CentOS 7 servers unusable: there is kernel panic as soon as the boot starts. I tried to start previous kernels from GRUB but all of them crash the same way. I even tried to install the latest 4.1 kernel, it also crashes with

kernel panic-not syncing: VFS: unable to mount root fs

There is an exception, though: the last line on GRUB which is the "Rescue kernel". This one boots fine, I can login on the console (there are no error messages displayed).

I tried to reinstall the kernels (yum reinstall kernel, yum remove kernel ; yum update) - same crash. There is about 200 MB left on /boot.

Where should I go from there? (short of reinstalling the OS)

EDIT: further information following comments

This is a physical machine, the filesystems are not encrypted

# rpm -qa kernel
kernel-3.10.0-123.13.2.el7.x86_64
kernel-3.10.0-229.7.2.el7.x86_64

# df -T
Filesystem                       Type     1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_euextscan-lv_root ext4      51475068 17826932  31010312  37% /
devtmpfs                         devtmpfs  16347896        0  16347896   0% /dev
tmpfs                            tmpfs     16367276        0  16367276   0% /dev/shm
tmpfs                            tmpfs     16367276    16932  16350344   1% /run
tmpfs                            tmpfs     16367276        0  16367276   0% /sys/fs/cgroup
/dev/sda1                        ext4        487652   179742    278214  40% /boot
/dev/mapper/vg_euextscan-lv_home ext4     507841520 11176144 470845424   3% /home

# vgdisplay
  --- Volume group ---
  VG Name               vg_euextscan
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               557.88 GiB
  PE Size               4.00 MiB
  Total PE              142818
  Alloc PE / Size       142818 / 557.88 GiB
  Free  PE / Size       0 / 0
  VG UUID               JOZX35-8T48-Yd10-wcta-wfSR-Fpf0-Hfkk1S

cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_euextscan-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sdb
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS Linux (3.10.0-229.7.2.el7.x86_64) 7 (Core)
        root (hd0,0)
        kernel /vmlinuz-3.10.0-229.7.2.el7.x86_64 ro root=/dev/mapper/vg_euextscan-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=fr LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_euextscan/lv_swap rd_LVM_LV=vg_euextscan/lv_root rd_NO_DM rhgb quiet
title CentOS Linux (4.1.0-1.el7.elrepo.x86_64) 7 (Core)
        root (hd0,0)
        kernel /vmlinuz-4.1.0-1.el7.elrepo.x86_64 ro root=/dev/mapper/vg_euextscan-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=fr LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_euextscan/lv_swap rd_LVM_LV=vg_euextscan/lv_root rd_NO_DM rhgb quiet
title CentOS Linux (3.10.0-123.13.2.el7.x86_64) 7 (Core)
        root (hd0,0)
        kernel /vmlinuz-3.10.0-123.13.2.el7.x86_64 ro root=/dev/mapper/vg_euextscan-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=fr LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_euextscan/lv_swap rd_LVM_LV=vg_euextscan/lv_root rd_NO_DM rhgb quiet
        initrd /initramfs-3.10.0-123.13.2.el7.x86_64.img
title CentOS Linux 7 Rescue 3df14aca2793db41f468b5f800000019 (3.10.0-123.9.2.el7.x86_64)
        root (hd0,0)
        kernel /vmlinuz-0-rescue-3df14aca2793db41f468b5f800000019 ro root=/dev/mapper/vg_euextscan-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=fr LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_euextscan/lv_swap rd_LVM_LV=vg_euextscan/lv_root rd_NO_DM rhgb quiet
        initrd /initramfs-0-rescue-3df14aca2793db41f468b5f800000019.img

Sorry, I forgot about that : yes, there is space left on `/boot` (about 200 MB) — WoJ, Jun 29 '15 at 16:54
More information is required: physical or virtual? how is your root filesystem configured? is there hardware or software RAID involved? LUKS encryption? have you already tried to inspect the filesystems from the rescue kernel? — dawud, Jun 29 '15 at 19:11
Also, a full grub configuration and list of disks/filesystems would surely help — stoned, Jun 29 '15 at 20:30
Did you revert to an older kernel? By default yum keeps several around, `yum list kernel`, pick a previous one and install it with `yum install kernel-`. — Brian, Jun 30 '15 at 11:50
@Brian: as I mentioned, none of the previous kernels worked (they crash the same way), except for the rescue one. — WoJ, Jun 30 '15 at 11:52
Where did you get a `/boot/grub/grub.conf` file from? You should not have this file at all. The system should be using grub 2 to boot. — Michael Hampton, Jun 30 '15 at 13:23
@MichaelHampton: the file is straight from the server, without modifications. — WoJ, Jun 30 '15 at 15:05
That's just it, _it should not exist_. Someone has done something very strange to your installation. — Michael Hampton, Jun 30 '15 at 15:08
@MichaelHampton: The [documentation](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/ch-Working_with_the_GRUB_2_Boot_Loader.html) states however that *GRUB 2 reads its configuration from the /boot/grub2/grub.cfg file on traditional BIOS-based machines and from the /boot/efi/EFI/redhat/grub.cfg file on UEFI machines.* — WoJ, Jul 01 '15 at 07:13

how to turn a rescue kernel into a working one?

0 Answers0