How to ensure sshd is the last service to be stopped during shutdown?

Question

I have noticed that when I issue a reboot command, my SSH connection is immediately closed on CentOS 7. It does not mean the server has rebooted, as I can still ping it.

It seems systemd is killing sshd too early. Sometimes a service will get stuck shutting down or just be slow, and I can't login anymore to verify what is wrong even though the server is up.

Is there a way to tell systemd to only kill sshd after all other service are down?

Not an answer to your question, but an unhelpful reminder: all production servers should have KVM, remote access card, or serial cable access. (though depending on where the shutdown is stalling, even those might be useless) — Andrew B, May 11 '15 at 14:04
This will not be easy; systemd is aggressively parallel with startup and shutdown. Not only is ssh being killed immediately, so are many other services, probably including the service which is hanging. Get on the (remote) console and have a look, or check the logs at next boot. — Michael Hampton, May 11 '15 at 16:17
Maybe the good folks at http://unix.stackexchange.com/ would have better advice for modifying the systemd shutdown process? — austinian, Jul 16 '15 at 19:43
It seems that systemd uses some ordering based on startup configuration for shutdown ordering: http://serverfault.com/questions/618612/how-to-sequence-a-proper-shutdown-and-startup-of-web-services-with-systemd but the reference is vague and long, I think they are pointing you towards DefaultDependencies - it seems to be saying that if it needs something to start up, at shutdown a service won't be shut down before the thing it depends on. — Mary, Jul 27 '15 at 22:47

score 1 · Answer 1 · answered Feb 11 '16 at 12:22

Even if you could ensure that SSH is the last service to be shut down, it would be pointless, because you will lose connection as soon as network services are stopped.

A Linux machine that boots starts all services in a pre-determined order: first the most vital services, the syslog, the firewall, then the network, and eventually the applications requiring network access (server web, server SSH, etc.). When shutting down, these services are stopped in reverse order. So you will lose SSH connection almost immediately.

The fact that SSH is already down while you can still ping the remote machine is normal.

score 0 · Answer 2 · edited Jul 16 '15 at 18:38

This is a bit of a hack, but what about adding something like:

ExecStopPre=sleep 60

to the "[Service]" section of /usr/lib/systemd/system/sshd.service? That wouldn't guarantee the ordering of the shutdowns, but if you could tolerate a longer shutdown time, it might allow you to keep your shell active long enough to verify that everything else terminated ok, or to do some quick poking around. Not sure if this is helpful in the general case, as it's probably too late to abort a problematic shutdown at this stage anyway, but it could be useful in a situation where you have a recurring problem during shutdown that you're trying to debug.

ostendali · Answer 3 · 2015-10-07T09:48:59.467

-5

I am not sure what you are after here but the first thing one will think about is to renice the process.

So in your case you can renice the sshd parent process:

       renice PRIO_MAX -p pid-of-sshd

edited Oct 07 '15 at 09:48

answered Oct 05 '15 at 15:49

ostendali

373
1
4

That will change its priority in CPU scheduling to near-realtime, but nothing else. – Falcon Momot Oct 05 '15 at 18:54

How to ensure sshd is the last service to be stopped during shutdown?

3 Answers3