105

So, say I get disconnected from an SSH-session after I've started rsync or cp or any other command that can be long running. Does that command keep running until it's finished after I get disconnected or does it just get killed?

Always wondered this.

Andrew B
  • 31,858
  • 12
  • 90
  • 128
fregas
  • 1,201
  • 2
  • 9
  • 7
  • I just want to add to what has been said above that, if you find yourself in a situation when you need to put an already running process into `screen`, try [reptyr](https://github.com/nelhage/reptyr). – a sad dude Jan 07 '13 at 19:00
  • It will kill your programs. Very annoying if you're updating OS version from say Ubuntu 16.04 to 17.10 via SSH – kurdtpage Aug 05 '19 at 04:45

3 Answers3

127

Edit for 2016:

This Q&A predates the systemd v230 debacle. As of systemd v230, the new default is to kill all children of a terminating login session, regardless of what historically valid precautions were taken to prevent this. The behavior can be changed by setting KillUserProcesses=no in /etc/systemd/logind.conf, or circumvented using the systemd-specific mechanisms for starting a daemon in userspace. Those mechanisms are outside the scope of this question.

The text below describes how things have traditionally worked in UNIX designspace for longer than Linux has existed.


They will get killed, but not necessarily immediately. It depends on how long it takes for the SSH daemon to decide that your connection is dead. What follows is a longer explanation that will help you understand how it actually works.

When you logged in, the SSH daemon allocated a pseudo-terminal for you and attached it to your user's configured login shell. This is called the controlling terminal. Every program you start normally at that point, no matter how many layers of shells deep, will ultimately trace its ancestry back to that shell. You can observe this with the pstree command.

When the SSH daemon process associated with your connection decides that your connection is dead, it sends a hangup signal (SIGHUP) to the login shell. This notifies the shell that you've vanished and that it should begin cleaning up after itself. What happens at this point is shell specific (search its documentation page for "HUP"), but for the most part it will start sending SIGHUP to running jobs associated with it before terminating. Each of those processes, in turn, will do whatever they're configured to do on receipt of that signal. Usually that means terminating. If those jobs have jobs of their own, the signal will often get passed along as well.

The processes that survive a hangup of the controlling terminal are ones that either disassociated themselves from having a terminal (daemon processes that you started inside of it), or ones that were invoked with a prefixed nohup command. (i.e. "don't hang up on this") Daemons interpret the HUP signal differently; since they do not have a controlling terminal and do not automatically receive a HUP signal, it is instead repurposed as a manual request from the administrator to reload the configuration. Ironically this means that most admins don't learn the "hangup" usage of this signal for non-daemons until much, much later. That's why you're reading this!

Terminal multiplexers are a common way of keeping your shell environment intact between disconnections. They allow you to detach from your shell processes in a way that you can reattach to them later, regardless of whether that disconnection was accidental or deliberate. tmux and screen are the more popular ones; syntax for using them is beyond the scope of your question, but they're worth looking into.


It was requested that I elaborate on how long it takes for the SSH daemon to decide that your connection is dead. This is a behavior which is specific to every implementation of a SSH daemon, but you can count on all of them to terminate when either side resets the TCP connection. This will happen quickly if the server attempts to write to the socket and the TCP packets are not acknowledged, or slowly if nothing is attempting to write to the PTY.

In this particular context, the factors most likely to trigger a write are:

  • A process (typically the one in the foreground) attempting to write to the PTY on the server side. (server->client)
  • The user attempting to write to the PTY on the client side. (client->server)
  • Keepalives of any sort. These are usually not enabled by default, either by the client or the server, and there are typically two flavors: application level and TCP based (i.e. SO_KEEPALIVE). Keepalives amount to either the server or the client infrequently sending packets to the other side, even when nothing would otherwise have a reason to write to the socket. While this is typically intended to skirt firewalls that time out connections too quickly, it has the added side effect of causing the sender to notice when the other side isn't responding that much more quickly.

The usual rules for TCP sessions apply here: if there is an interruption in connectivity between the client and server, but neither side attempts to send a packet during the problem, the connection will survive provided that both sides are responsive afterwards and receiving the expected TCP sequence numbers.

If one side has decided that the socket is dead, the effects are typically immediate: the sshd process will send HUP and self-terminate (as described earlier), or the client will notify the user of the detected problem. It's worth noting that just because one side thinks the other is dead does not mean that the other is has been notified of this. The orphaned side of the connection will typically remain open until either it attempts to write to it and times out, or receives a TCP reset from the other side. (if connectivity was available at the time) The cleanup described in this answer only happens once the server has noticed.

Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • 3
    I'd also add the 'dtach' command to that list - it's sort of the opposite of screen/tmux in that it lets you attach multiple terminals to a single session, and is also great for prolonging a session, although it doesn't provide any means of replaying the recent history. – fluffy Jan 06 '13 at 17:10
  • `dtach` url is here: http://dtach.sourceforge.net/ – slm Jan 06 '13 at 20:17
  • 2
    excellent explanation. I feel more linuxey already! – fregas Jan 07 '13 at 22:31
  • 3
    Also, while it's not technically part of your answer, here's an interesting bit of trivia: you can use `kill -HUP` as root to force someone's terminal to hang up. You shouldn't do this without good reason. I get most of my mileage out of it when users leave shells running during maintenance and I need to unmount a filesystem that their shell is keeping open. If the user is connected but terminally idle, send the signal to their sshd process. Otherwise, if it's running inside of a terminal multiplexer, send it to the shell you want stopped. Only hang up the shell keeping you from working! – Andrew B Jan 07 '13 at 23:36
  • @AndrewB, could you please elaborate on how "the SSH daemon process ... decides that your connection is dead"? And how can I know if the SSH daemon thinks/knows that the (which) connection is dead or not? – Devs love ZenUML Apr 11 '16 at 02:02
  • @Xiao Edited the answer to clarify, see the bottom. Beyond these generalizations, you will need to either consult the documentation or examine the source code for the product in question. – Andrew B Apr 11 '16 at 02:37
  • This is an impressive and well written answer. – Aaron May 26 '17 at 12:17
  • Please clarify, tmux is work as expected (not killed by systemd)? – mmv-ru Mar 23 '18 at 23:18
  • @mmv-ru Beyond the scope of my ability to answer. I have no idea what version of systemd you are running, how it has been built, or how it has been configured. [tmux refused to change themselves for systemd](https://github.com/tmux/tmux/issues/428), so my assumption is that it still gets killed unless your distro changed the default settings. (edit: As an example, [Fedora is currently overriding the default](https://bugzilla.redhat.com/show_bug.cgi?id=1357426).) – Andrew B Mar 24 '18 at 00:11
25

As other's have mentioned, once you disconnect from ssh anything running within it is gone.

As @Michael Hampton and others have mentioned you can use tools like tmux or screen to disconnect/reconnect to terminals without loosing their content (i.e. child processes).

Additionally you can put a process into the background using an ampersand & and then use the command disown to disassociate them with the current shell.

# start a command
% sleep 5000 &
[1] 3820

# check it
% jobs
[1]+  Running                 sleep 5000 &

# disown everything
% disown -a

# check it again (gone from shell)
% jobs
%

# but it's still running on the system
% ps -eaf|grep "[s]leep"
saml      3820 23791  0 00:16 pts/1    00:00:00 sleep 5000
%
slm
  • 7,355
  • 16
  • 54
  • 72
  • Is it possible to reattach a terminal session to a `disown`ed process? – Fake Name Jan 06 '13 at 07:25
  • 4
    Yes. See this U&L question for details: http://unix.stackexchange.com/questions/4034/how-can-i-disown-a-running-process-and-associate-it-to-a-new-screen-shell – slm Jan 06 '13 at 07:33
15

No, any programs still attached to the terminal, and not placed into the background with something like nohup, would be killed.

This is why there are virtual terminal solutions like tmux and the older screen which create sessions which continue running even if you are disconnected, and to which you can reattach later.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940