0

I have a specific situation with one remote Debian(Stretch) server. Mainly there is no one at the site and I need to do some risky upgrades and reconfigurations remotely.

I'm looking for a way to implement a fail-safe system to prevent the situation where I lock myself out. For example, if after a reboot I'm not able to log back in via ssh for a set amount of time that the system issues a dd restore on the root partition from a previously done dd clone.

The main problem is how to do the dd clone/restore safely on a running system. I tried it on my test rig... and it ends up not booting with "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY." message

Thankfully fsck was able to fix the issues and make the system bootable again... But this seems to me more just luck than a proper method to do it... On top of it, I was able to run fsck manually only because I was physically there next to the machine, which I wouldn't be able to do remotely(maybe it could be scripted)

Is there a way to unmount / and do the clone... Downtime is not an issue.. So a solution which would be automated via a offline script is totally acceptable. I was thinking maybe there is a way to script dd to run at the very last stage of a shutdown/reboot process, when the partition unmounts, but will dd be available then? :-/

Before I go on reinventing the wheel, I though I better ask here for a proven method. Thank you all for any help!

Ivan Kovacevic
  • 1,671
  • 3
  • 14
  • 19
  • Boot from an installation or rescue media. Stop using ext4 for filesystems. Use XFS instead. It is much more resilient to this sort of thing. – Michael Hampton Aug 05 '20 at 18:55
  • I would, if I was able to go to the machine and plug a usb drive or a DVD.... But since I can't get to the machine I'm looking for a remote solution. Maybe It would be possible to create a dual boot environment? So that I remotely reboot into the other system – Ivan Kovacevic Aug 05 '20 at 18:57
  • Use the remote console (IPMI, iDRAC, ILO, etc) – Michael Hampton Aug 05 '20 at 18:59
  • Unfortunately I don't have any at disposal. This remote server can barely be called that.. As it is just a home desktop PC(headless) acting as a server, with no additional special networking hardware... Is there a way to add a remote console just via software, that will run at boot time? – Ivan Kovacevic Aug 05 '20 at 19:07
  • You can hook up a KVM switch but that also requires its own network access, and someone to physically go there and do it. I expect you are going to have to visit the site. – Michael Hampton Aug 05 '20 at 19:18
  • Yup I definitely plan on adding some hardware the next time I visit the site, which would render this problem moot. Meanwhile I'm looking if there is some half-sketchy way of doing it with what I have now... Thanks for all the advices(XFS, console...)! – Ivan Kovacevic Aug 05 '20 at 19:24
  • 1
    If you have some spare hard disk space (doesn't have to be a lot), you could debootstrap a basic failsafe root partition -- then if something goes awry during your maintenance, have some grub reconfiguration scripted to boot with the failsafe partition as root instead of trying to dd over an existing root. – Brandon Xavier Aug 06 '20 at 06:19
  • Brandon, I like that idea! Thanks. I started researching the details on how to implement it... – Ivan Kovacevic Aug 06 '20 at 18:52

0 Answers0