0
This morning a user came up to me to let me know that email being sent from the outside world was not showing up in his inbox for his work email. We host the email server internally (not my idea). I tested by sending an email from my gmail account to my work email, and as expected the email never arrives.
I go to reboot the mail server (which is running Ubuntu 10.10) and right before the point where the GRUB boot loader is started, the server would reboot back to a POST/BIOS "Checking memory" message as if I had hit the reset button.
I went and burned a copy of 10.10 and tried to run the system rescue to see if I was able to fix the boot issue, but I was not able to fix it. I then burned a copy of Ubuntu 18.04 to see if there were any additional tools that might allow me to dig a little deeper to find out what was going on.
I checked /etc/fstab on the drive where Ubuntu is currently installed, and here is what I found
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
/dev/mapper/mailadmin-root / ext4 errors=remount-ro 0 1
# /boot was on /dev/sda2 during installation
UUID=e8c63c19-db49-4013-bec8-ef8479fb52f1 /boot ext2 defaults 0 2
/dev/mapper/mailadmin-swap_1 none swap sw 0 0
10.6.9.5:/volume1/Backups/Zimbra/TarBackups /zmbackup nfs,size=8192,wsize=8192,timeo=14,intr
Notice the comment above the UUID for the /boot partition? I did not enter that comment, it was left by the previous IT manager a number of years ago. I modified the /etc/fstab file to point at /dev/sda2 (which I have confirmed is the actual location where the grub config is located) instead of the UUID, rebooted - but still the server goes into a reboot loop right as it is about to load grub.
After doing some googling, I found a script called bootinfoscript.sh that outputs a text file with a bunch of information related to the hard disks that are detected in my system. Here is the output of bootinfoscript.sh
Boot Info Script 0.61 [1 April 2012]
============================= Boot Info Summary: ===============================
=> Grub2 (v1.97-1.98) is installed in the MBR of /dev/sda and looks at sector
2048 of the same hard drive for core.img. core.img is at this location and
looks in partition 2 for /grub.
=> Syslinux MBR (4.04 and higher) is installed in the MBR of /dev/sdb.
sda1: __________________________________________________________________________
File system: BIOS Boot partition
Boot sector type: Grub2's core.img
Boot sector info:
sda2: __________________________________________________________________________
File system: ext2
Boot sector type: -
Boot sector info:
Operating System:
Boot files: /grub/grub.cfg /grub/core.img
sda3: __________________________________________________________________________
File system: LVM2_member
Boot sector type: -
Boot sector info:
sdb1: __________________________________________________________________________
File system: vfat
Boot sector type: SYSLINUX 6.03
Boot sector info: Syslinux looks at sector 32784 of /dev/sdb1 for its
second stage. The integrity check of Syslinux failed.
No errors found in the Boot Parameter Block.
Operating System:
Boot files: /boot/grub/grub.cfg /syslinux.cfg
/efi/BOOT/grubx64.efi /ldlinux.sys
mailadmin-root': _______________________________________________________________
File system:
Boot sector type: Unknown
Boot sector info:
Mounting failed: mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-root': unknown filesystem type ''.
mailadmin-swap_1': _____________________________________________________________
File system:
Boot sector type: Unknown
Boot sector info:
Mounting failed: mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-root': unknown filesystem type ''.
mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-swap_1': unknown filesystem type ''.
============================ Drive/Partition Info: =============================
Drive: sda _____________________________________________________________________
Disk /dev/sda: 10.9 TiB, 11999064883200 bytes, 23435673600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Partition Boot Start Sector End Sector # of Sectors Id System
/dev/sda1 1 4,294,967,295 4,294,967,295 ee GPT
GUID Partition Table detected.
Partition Start Sector End Sector # of Sectors System
/dev/sda1 2,048 4,095 2,048 BIOS Boot partition
/dev/sda2 4,096 503,807 499,712 Data partition (Windows/Linux)
/dev/sda3 503,808 7,329,544,191 7,329,040,384 Logical Volume Manager (LVM) partition (Linux)
Drive: sdb _____________________________________________________________________
Disk /dev/sdb: 14.4 GiB, 15500574720 bytes, 30274560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Partition Boot Start Sector End Sector # of Sectors Id System
/dev/sdb1 * 2,048 30,274,559 30,272,512 c W95 FAT32 (LBA)
"blkid" output: ________________________________________________________________
Device UUID TYPE LABEL
/dev/loop0 squashfs
/dev/loop1 squashfs
/dev/loop2 squashfs
/dev/loop3 squashfs
/dev/loop4 squashfs
/dev/loop5 squashfs
/dev/loop6 squashfs
/dev/loop7 squashfs
/dev/mapper/mailadmin-root 3e2b9e3a-7190-4240-804e-f1761fa96ade ext4
/dev/mapper/mailadmin-swap_1 2b7f558a-2abe-4981-879e-d79f7ed3fb01 swap
/dev/sda1
/dev/sda2 e8c63c19-db49-4013-bec8-ef8479fb52f1 ext2
/dev/sda3 WFe0Xa-LXJa-BzOr-Oyo9-kC5E-3W7p-qWA5Uy LVM2_member
/dev/sdb1 6045-3252 vfat UBUNTU 18_0
========================= "ls -R /dev/mapper/" output: =========================
/dev/mapper:
control
mailadmin-root
mailadmin-swap_1
================================ Mount points: =================================
Device Mount_Point Type Options
/dev/loop0 /rofs squashfs (ro,noatime)
/dev/sdb1 /cdrom vfat (ro,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
============================= sda2/grub/grub.cfg: ==============================
--------------------------------------------------------------------------------
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by /usr/sbin/grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
load_env
fi
set default="0"
if [ ${prev_saved_entry} ]; then
set saved_entry=${prev_saved_entry}
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi
function savedefault {
if [ -z ${boot_once} ]; then
saved_entry=${chosen}
save_env saved_entry
fi
}
function recordfail {
set recordfail=1
if [ -n ${have_grubenv} ]; then if [ -z ${boot_once} ]; then save_env recordfail; fi; fi
}
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1
set locale_dir=($root)/grub/locale
set lang=en
insmod gettext
if [ ${recordfail} = 1 ]; then
set timeout=-1
else
set timeout=10
fi
### END /etc/grub.d/00_header ###
### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###
### BEGIN /etc/grub.d/10_linux ###
### END /etc/grub.d/10_linux ###
### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1
linux16 /memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1
linux16 /memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###
### BEGIN /etc/grub.d/30_os-prober ###
if [ ${timeout} != -1 ]; then
if keystatus; then
if keystatus --shift; then
set timeout=-1
else
set timeout=0
fi
else
if sleep --interruptible 3 ; then
set timeout=0
fi
fi
fi
### END /etc/grub.d/30_os-prober ###
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###
--------------------------------------------------------------------------------
=================== sda2: Location of files loaded by Grub: ====================
GiB - GB File Fragment(s)
=========================== sdb1/boot/grub/grub.cfg: ===========================
--------------------------------------------------------------------------------
if loadfont /boot/grub/font.pf2 ; then
set gfxmode=auto
insmod efi_gop
insmod efi_uga
insmod gfxterm
terminal_output gfxterm
fi
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
set timeout=5
menuentry "Try Ubuntu without installing" {
set gfxpayload=keep
linux /casper/vmlinuz file=/cdrom/preseed/ubuntu.seed boot=casper quiet splash ---
initrd /casper/initrd.lz
}
menuentry "Install Ubuntu" {
set gfxpayload=keep
linux /casper/vmlinuz file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---
initrd /casper/initrd.lz
}
menuentry "OEM install (for manufacturers)" {
set gfxpayload=keep
linux /casper/vmlinuz file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash oem-config/enable=true ---
initrd /casper/initrd.lz
}
menuentry "Check disc for defects" {
set gfxpayload=keep
linux /casper/vmlinuz boot=casper integrity-check quiet splash ---
initrd /casper/initrd.lz
}
--------------------------------------------------------------------------------
============================== sdb1/syslinux.cfg: ==============================
--------------------------------------------------------------------------------
DEFAULT loadconfig
LABEL loadconfig
CONFIG /isolinux/isolinux.cfg
APPEND /isolinux/
--------------------------------------------------------------------------------
=================== sdb1: Location of files loaded by Grub: ====================
GiB - GB File Fragment(s)
================= sdb1: Location of files loaded by Syslinux: ==================
GiB - GB File Fragment(s)
======================== Unknown MBRs/Boot Sectors/etc: ========================
Unknown BootLoader on mailadmin-root'
Unknown BootLoader on mailadmin-swap_1'
=============================== StdErr Messages: ===============================
cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory
cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory
cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory
Failed to find logical volume "mailadmin/root'"
Failed to find logical volume "mailadmin/root'"
Failed to find logical volume "mailadmin/root'"
hexdump: /dev/mapper/mailadmin-root': No such file or directory
hexdump: /dev/mapper/mailadmin-root': No such file or directory
Failed to find logical volume "mailadmin/swap_1'"
Failed to find logical volume "mailadmin/swap_1'"
Failed to find logical volume "mailadmin/swap_1'"
hexdump: /dev/mapper/mailadmin-swap_1': No such file or directory
hexdump: /dev/mapper/mailadmin-swap_1': No such file or directory
Just for some background info, /dev/sda is a 12TB partition where the mail server software is installed. /dev/sdb is a 16GB USB thumbdrive that I installed Ubuntu onto.
I mounted the 12TB (which is actually a RAID array) /dev/sda1 drive and when I browse to the /boot folder on the mounted drive, it appears as if the /boot directory is completely empty!
Anyone have any idea what is going on here? Is there some way to fix the grub boot loader on this old server?
Please let me know if you need any additional details.
Richie086
Posted 2018-12-28T22:36:26.993
Reputation: 4 299
And what do you see if you mount
/dev/sda2? The fact thatbootis empty in the screenshot is no surprise, it would only be populated if you manually mounted it, too, at this stage. – tink – 2018-12-29T00:28:37.0671
You desperately need to update your server to a supported version of Ubuntu. Ubuntu 10.10 reached end-of-life in April 2012, and has not received any updates (including security updates!!) since then.
– duskwuff -inactive- – 2019-02-08T02:44:44.333@duskwuff tell me about it! I took over as the IT manager at this company 2 years ago and I’ve replaced everything except for this last server. It had 10 years worth of archived email for 75 users, and was expected to be up 100% of the time by upper management. We are in the process of migrating everything over to G Suite as I write this comment :) – Richie086 – 2019-02-08T02:49:49.557