Attach to the same tmux session from different nodes of a cluster

Question

I'm working on a computing cluster which has several hundreds of nodes. A load-balancing scheme picks an idle node to login to when ssh-ing into the cluster. As a result, it is very unlikely to re-login into a previous machine on subsequent calls to ssh.

However, I would like to use tmux on the remote to setup a session which persists between different logins.

I'm looking for different options which can be used to achieve the desired behaviour. I came up with the following option, which is, however, not really a "good" solution:

• Pick one node and stick to it.
  I.e. after being logged into machine node-XXXX by the load-balancer open a new ssh connection to node-0042 and run tmux there.

  Using ssh's ProxyCommand this could even be achieved in a seamless manner, i.e. the user types ssh node-0042 which opens a connection the the cluster login and uses this connection as proxy for a connection the node-0042.

  This has the obvious downside that the user does not gain anything from the load-balancing mechanism.

Do you have any advice on how to handle this situation? Any idea for a different approach to solve it (is it possible to "move" a running tmux server to a different machine [which is a carbon copy of the current machine]?).

Answer 1

It is not possible to "move" a tmux session, so you really need to get back to the same host if you want to use tmux.

I like to make use of SSH's multiplexing abilities. Once you establish the connection, it persists in the background, and when you try to ssh to the same host again, it will re-use the existing ssh connection, getting you back to the same node you were on before.

Setting this up is pretty easy in your ~/.ssh/config:

Host cluster
  Hostname cluster.local
  ForwardAgent yes
  ControlPath ~/.ssh/ctl-%r@%h:%p
  ControlMaster auto
  ControlPersist yes

(There are more options for this, see your local man ssh_config for all of them)

It does not solve the problem if your connection actually goes away (put your workstation to sleep, long periods of idle, etc), but can be really helpful for short-term considerations.

Answer 2

You might set up a service address for your shell machine, so a DNS entry can point to the wherever it is. With some kind of dynamic DNS service or high availability software, you can use the same host name every time.

If you are able to run VMs on top of these resources, you could alternately boot your favorite OS just to run your shell and attach to tmux (running ssh sessions or whatever). The static address would be to this shell VM. If you have live migrate capability, it can even keep running when some nodes have to be taken down for maintenance.