SSH broken pipe kills foreground process

Question

I have 2 hosts that I can connect them through SSH. In one host, after a while, I get the error Write Failed: broken pipe. In the other host, I don't get this error.

I don't understand why in one host, I get this problem, and in another host everything is ok.

So, I have set a SSH connection with tmux, and inside the remote host I run a foreground process (celeryd). But, even when the SSH session fails, and I reconnect to the dropped session, the foreground process has died.

I would like that the foreground process won't come down when the SSH connection is broken. So, I just have 2 solutions:

1. know how to avoid SSH broken pipes, or
2. know how to avoid that the process will die when the SSH connection goes down.

Can anyone help me on this?

Answer 1

I don't know what happens with your SSH connection, but probably a bad network link? You can try using MOSH, which is more robust than a regular SSH connection. Especially if the network is the problem.

If you can not fix the connection, you could try using nohup in front of your command or just nohup tmux and see if that changes anything.

nohup - run a command immune to hangups, with output to a non-tty

SYNOPSIS
nohup COMMAND [ARG]...
nohup OPTION

Answer 2

I have run across this issue only a few times. For me what would cause the broken pipe was a missing route. It only occurred when connecting from a physical machine on the main network to a VM on a different network behind a Nat.

I would check for networking issues to see if you can stop the broken pipes from occuring.

If you can not find a cause for the issue, you can use something like screen or tmux to keep a terminal open even if you are disconnected. Then you can reconnect and connect to the screen or tmux session and your process will still be running and active.

Answer 3

Most common cause for ssh disconnects for me has been from no keep-alives. It is settable in the ssh client. With ssh, it would be "-oTCPKeepAlives=yes"

with regard to programs running after the ssh has closed, when you run a program in the shell, it is a child process of the shell instance. Once it closes, the children close, too. You can try running your program with a & at the end which would spawn off a different process. I just did a quick test and the process seems to continue after the ssh connection is disconnected.