At our startup, while trying to move our applications to a new server we ran into a hung up Ubuntu 16.04 at the new server, responding to pings but ssh hanging at
debug1: Entering interactive session.
and no login showing up when trying to login with screen + keyboard directly.
After power cycling the server it booted without any errors, but some (or all?) files that have been opened by applications like logfiles had some blocks of zeros at their end, so the files seem to be corrupted.
Our disk setup is as follows:
3 x SSD, configured as Software Raid 5 (mdadm) with LVM on top:
- 1x ext4 Logical Volume for Host OS(Ubuntu 16.04)
- 1x ext4 Logical Volume holding mysql datadir used from a Virtual Machine
3 x HDD, configuredd as Software Raid 5 (mdadm) with LVM on top.
- Raid is congigured for 4 disks, with one missing that we'll add later
- 1x ext4 Logical Volume for data storage
Server Configuration:
384 GB Ram
2x Xeon E5-2620 v4
My questions are:
- Are the files currupted in a way that we need to use a backup or shall we continue operation?
- How could a fresh installed system hang after such a short period of uptime?
My guesses are:
- files maybe currupted, so we should use a backup
- the crash may have occured due to the os file system cache filling up rapidly and a possible misalignment of the raid+lvm+virtual machine setup that made the SSDs too slow to keep up with i/o resulting in a frozen system