I performed a system upgrade from CentOS 7 to AlmaLinux 8 rather successfully (as noted here). During the process something strange happened and GRUB lost its boot entries entirely and my limited efforts in rebuilding them / it (I really only need the AlmaLinux 8 kernel entry and there's only one vmlinuz + initrd image so far from the update) have met failure (especially since I've not had to fiddle with GRUB directly since my Gentoo days a couple decades ago).
Presently in order to get the server to boot I essentially have to go to the LISH (Glish specifically) console on the Linode server and this is what I get on reboot:
Glish:
SeaBIOS (version rel-1.12.1-0.ga5ca58e9aef-prebuilt.qemu.org)
iPXE (http://ipxe.org) 00:04.0 C980 PCI2.10 PnP PMM+7FF90F20+7FEF0F20 C980
Booting from ROM...
Welcome to GRUB!
error: variable `prefix' isn't set.
Then in Lish & Weblish:
GNU GRUB version 2.04
Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists possible
device or file completions.
grub> _
I then have to enter something along the following lines, which I threw together myself as a desperate effort via ls (hd0)/boot/ to just get the system booting into non-single user mode, to get it to boot:
set root=(hd0)
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda console=ttyS0,19200n8
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
boot
( A perhaps refined set of boot args that should be issued were found later following investigation and discussion:
set root=(hd0)
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
boot
)
I figured I could simply repair this with grub2-mkconfig once in via the following command, but as you can see, it simply didn't find any entries:
$ grub2-mkconfig --output /boot/grub2/grub.cfg
Generating grub configuration file ...
done
When looking into the generated b2/grub.cfg :: 10_linux section we see:
### BEGIN /etc/grub.d/10_linux ###
insmod ext2
set root='hd1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0 --hint-efi=hd0 --hint-baremetal=ahci0 --hint='hd1' 06e9a827-47b4-43f9-8e01-91b3852f86a8
else
search --no-floppy --fs-uuid --set=root 06e9a827-47b4-43f9-8e01-91b3852f86a8
fi
insmod ext2
set boot='hd1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=boot --hint-bios=hd0 --hint-efi=hd0 --hint-baremetal=ahci0 --hint='hd1' 06e9a827-47b4-43f9-8e01-91b3852f86a8
else
search --no-floppy --fs-uuid --set=boot 06e9a827-47b4-43f9-8e01-91b3852f86a8
fi
# This section was generated by a script. Do not modify the generated file - all changes
# will be lost the next time file is regenerated. Instead edit the BootLoaderSpec files.
#
# The blscfg command parses the BootLoaderSpec files stored in /boot/loader/entries and
# populates the boot menu. Please refer to the Boot Loader Specification documentation
# for the files format: https://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/.
# The kernelopts variable should be defined in the grubenv file. But to ensure that menu
# entries populated from BootLoaderSpec files that use this variable work correctly even
# without a grubenv file, define a fallback kernelopts variable if this has not been set.
#
# The kernelopts variable in the grubenv file can be modified using the grubby tool or by
# executing the grub2-mkconfig tool. For the latter, the values of the GRUB_CMDLINE_LINUX
# and GRUB_CMDLINE_LINUX_DEFAULT options from /etc/default/grub file are used to set both
# the kernelopts variable in the grubenv file and the fallback kernelopts variable.
if [ -z "${kernelopts}" ]; then
set kernelopts="root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0 "
fi
insmod blscfg
blscfg
### END /etc/grub.d/10_linux ###
Also /boot/loader/entries/ does indeed have a file that seems to correspond to what should be booting via BLS:
title AlmaLinux (4.18.0-372.9.1.el8.x86_64) 8.6 (Sky Tiger)
version 4.18.0-372.9.1.el8.x86_64
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id almalinux-20220510130458-4.18.0-372.9.1.el8.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
And as for grubenv:
$ grub2-editenv list
saved_entry=4f09fa5fdd3642fa85221d7c11370603-4.18.0-372.9.1.el8.x86_64
kernelopts=root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0
boot_indeterminate=1
It seems like the GRUB entry during boot should contain:
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0
Yet it does not... And I haven't actually modified anything manually at this point and so I don't know where / how GRUB is getting confused.
As a sanity check, I verified packages are installed as expected:
$ dnf install kernel grub2
Last metadata expiration check: 2:31:00 ago on Fri Jun 17 13:43:41 2022.
Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed.
Package grub2-pc-1:2.02-123.el8.alma.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Ultimately it appears as though GRUB either doesn't find or fails to parse grub.cfg...
I thought that perhaps there may have been some kind of a change involving EFI since it seems that the update was happening at /boot/grub2/grub.cfg and I noticed a new EMPTY folder had been generated at /boot/efi/EFI/almalinux/ and so I tried generating the grub.cfg there as well:
$ grub2-mkconfig --output /boot/efi/EFI/almalinux/grub.cfg
Generating grub configuration file ...
done
But as you can see, there were still no entries in and during generation and when I rebooted, no BLS autogeneration/usage was performed either (as I think I understand it could have happened).
Here are the contents of various possibly pertinent /boot/ folders:
https://gist.github.com/ylluminate/d469ce8ace202a57f67f5aed255154cd
What can I do to get GRUB building and/or parsing and using a proper config again?
