How to make systemd start ssh.service before generated crypttab units

1

I have a server, running Debian Buster, whose root filesystem is not encrypted, but other drives are.
[ETA: All of the OS system directories (/var, /home, /etc, etc) are in the unencrypted root partition. Swap is encrypted, but it's created non-interactively with a new random key every boot, so nothing is held up waiting for that.]
I have entries in /etc/crypttab and /etc/fstab to decrypt and mount the encrypted 'storage' drives, and the boot process will wait indefinitely for the passphrase to the tiny encrypted partition that holds the keyfiles that decrypt the other drives. So far so good. The passphrase prompt comes up on a physically attached monitor, I type in the passphrase with a physically attached keyboard, and boot proceeds. I have OpenSSH daemon installed and use public-key authentication to connect to the server for all sorts of administration once it's booted.

Now I want to be able to remotely unlock those LUKS devices over ssh. Since root is not one of the encrypted filesystems, I don't see any need for the whole "dropbear+busybox in initramfs" business, and I thought this would be relatively straightforward.
sudo systemctl edit ssh.service and add the following lines:

[Unit]
Before=cryptsetup-pre.target
[Install]
WantedBy=cryptsetup-pre.target

This should make systemd shuffle things around to start sshd before starting any decryption of local disks, right? (I chose WantedBy rather than RequiredBy because I don't want systemd to 'Fail' cryptsetup if sshd doesn't come up as expected.)

sudo systemctl daemon-reload does not complain, and cat /etc/systemd/system/ssh.service.d/override.conf shows those four lines as expected. On reboot, the passphrase prompt comes up (expected), but all attempts to ssh in to the server come back with No route to host, so clearly ssh.service did not start first.

I tried changing the override to specify that ssh is WantedBy and comes Before systemd's generated systemd-cryptsetup@foo.service unit instead. Interestingly, this resulted in no ssh and no passphrase prompt, so boot was just hung forever and I had to boot from recovery media and remove the override. That's not exactly helpful, but at least it had an observable effect. Maybe I've created a circular dependency somewhere? The passphrase prompt does seem to come really early in the boot sequence.

What are my next steps here? Is it possible to override systemd's generated cryptsetup unit to give it Wants= and After= links to ssh.service? If it is possible, is there any reason to think that would get better results? Would it make sense to write my own units under /etc/ for that keystore partition and remove the corresponding inittab and fstab entries? In short, is there any remotely sane way to make this work, or should I give up on this avenue and go for dropbear+busybox in initramfs after all?

PS. I thought about just making the encrypted drives 'noauto', but I have VMs on those disks configured to autostart later in the boot process, so I would really rather get remote-unlocking working.

Askeli

Posted 2019-09-28T01:37:17.310

Reputation: 11

Are '/usr', '/var' and their respective nested directories among the encrypted disks/partitions if so this would explain the failure to start ssh as some of the needed buts for shhd (especially remote) are in those parts – linuxdev2013 – 2019-09-28T02:08:59.397

the boot process will wait indefinitely for the passphrase to the tiny encrypted partition that holds the keyfiles that decrypt the other drives I guess it prevents the service/daemon for network connectivity from starting as well? – Tom Yan – 2019-09-28T04:07:17.893

@TomYan: Well, yes, if it systemd's ordering starts the cryptsetup unit for that partition before the network target unit, then it makes sense that anything relying on network is not started yet either. But the question is how to make systemd start the network and ssh.service before that cryptsetup service instead. – Askeli – 2019-09-28T17:29:21.377

@linuxdev2013: Good question, but no. I used the "everything in one partition" option in the Debian installer, so all the 'system' directories (and even /home) are in the unencrypted root filesystem. I thought it would make life simpler for things like what I'm trying to do now! I have edited my question to include that detail. – Askeli – 2019-09-28T17:44:22.017

so clearly ssh.service did not start first what I meant was this might not be true. So all you need to do might be making whatever you use for network starts earlier like you did for sshd (btw was ssh.service a typo?). – Tom Yan – 2019-09-28T23:29:15.640

@TomYan: Ah, I see what you're getting at now. I was assuming that systemd was, um, not completely stupid, and the dependency chain would ensure that bringing up networking is a prerequisite for a network service like sshd. Maybe I need to specify a Requires there myself, though? – Askeli – 2019-09-29T02:14:50.150

Not sure. If the network daemon was brought up parallelly with sshd, I would probably just make it brought up by and before cryptsetup-pre.target. – Tom Yan – 2019-09-29T02:24:07.540

Grr. Didn't realize the Enter key in comments means "post", not "new line". And then my edit was rejected because I took more than five minutes... So anyway:

On examining ssh.unit, I see there's a line After=network.target. No Wants or Requires but AFAIK that shouldn't matter, because networking does start during bootup, and the After should ensure the right order, right?

Oddly, no typo. /etc/systemd/system/sshd.service exists but is just a symlink to /lib/systemd/system/ssh.service, which in turn calls sshd in its Exec* lines. – Askeli – 2019-09-29T02:28:18.053

No answers