11

I use a perl script (with the Semaphore package) to fire off long-running ssh commands to AWS instances. For several reasons, I do NOT run the ssh commands in the background.

Recently, the Comcast fiber to our office building was cut by a construction crew. We maintain a backup CenturyLink connection, and our IT people switched our office connection over to CenturyLink.

My ssh processes died with a "timeout, server not responding" message when we switched from Comcast to CenturyLink. They died again when we switched back to Comcast after the fiber was repaired.

Is this expected behavior for an open ssh command if the local public IP address changes? If I put the ssh commands in the background, would it solve this issue?

Morgan Brown
  • 153
  • 1
  • 10

3 Answers3

20

Yes, that's expected behavior. Your connection was physically interrupted, after all.

If an interruption in your connection may cause a failure of a long running remote process, consider running it in a tmux session.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • Thank you, Michael. I will research tmux. Do you recommend tmux over nohup? – Morgan Brown Jul 14 '20 at 14:32
  • 2
    @MorganBrown You would use tmux if you want to reconnect to the session. You can't do that with nohup. – Michael Hampton Jul 14 '20 at 14:35
  • 4
    Tmux is really very handy. It is well worth the time to learn it, particularly if you're running long-running jobs over `ssh`. I highly recommend it. – chicks Jul 15 '20 at 02:36
  • @MichaelHampton how this works? If connection is physically interrupted then ssh connection from local to aws will fail right? irrespective of how we are running, whether it be background(tmux) or normal? – lokanadham100 Jul 15 '20 at 02:38
  • @lokanadham100 https://github.com/tmux/tmux/wiki/Getting-Started – Michael Hampton Jul 15 '20 at 02:43
  • 1
    An IP address change is not a physical interruption. A physical interruption is unplugging a cable. IP addresses are not physical; physics does not apply to them. – user253751 Jul 15 '20 at 08:20
  • 10
    @user253751 The OP suffered a fiber cut and began using their backup ISP. I would consider that a physical interruption. – Michael Hampton Jul 15 '20 at 11:46
  • 1
    tmux or screen. – Gnudiff Jul 15 '20 at 20:46
  • FWIW, I work over SSH on a daily basis, using exactly tmux. – iBug Jul 16 '20 at 14:22
  • @user253751 They say IT people switched the connection, so it does not look like a transparent, automated switch, nor multipath TCP. – Didier L Jul 16 '20 at 16:10
  • @user253751 It sounds like you need to learn the basics of IP. Yes, the connection was interrupted, so they had to use another one. – Michael Hampton Jul 16 '20 at 16:17
  • @user253751 The only way that they could maintain the network connection in the event of a fiber cut is if they had their own PI addresses and were multihomed. That's not the setup they have here. So I'm apparently not understanding what you are trying to say. – Michael Hampton Jul 16 '20 at 16:22
14

If you do not want to run ssh commands in the background (manually) you may give mosh (mobile shell) a try. It's available for probably any distribution and system including various BSD, Android, Windows and macOS.

Beko Pharm
  • 306
  • 1
  • 6
  • I thought mosh was more for dealing with irritating latency. Does it handle losing the connection too? – chicks Jul 15 '20 at 02:37
  • 7
    @chicks yes, mosh ("MObile SHell") gracefully handles connection drops and even client IP changes (eg if the client is a mobile device switching cell towers). It makes use of UDP in order to do that. – vikarjramun Jul 15 '20 at 04:39
  • The traditional solution is also to run `screen` on the server. If your connection is lost, log back in and reconnect to the same `screen` by running `screen -r`. – user253751 Jul 15 '20 at 13:07
3

Yes, it's normal.

A TCP server identifies a connection by the combination of local address, local port, remote address and remote port.

Since you talk about "the local public IP address changes" I presume your workplace uses some form of NAT to map the private addresses on your LAN to one or more public addresses on the current internet connection.

Depending on how exactly the NAT is implemented there are several possibilities for what exactly might happen at the packet level, but for regular TCP all of them end up with the TCP connection failing.

There is an extension called "multipath TCP" which (among other things) allows connections to be maintained across network changes, but afaict using it between linux boxes still requires third party kernel modules at this time.

Peter Green
  • 4,056
  • 10
  • 29
  • 2
    Yes indeed. Another good reason for IPv6 here. If the company had been using an assigned IP block of their own and had a routing agreement with two ISPs, the switchover would have been transparent and no IP addresses would have changed. – Zan Lynx Jul 16 '20 at 17:08
  • 1
    That is really an advantage of going with provider-independent space, provider-independent space generally comes at a cost both to the company using the space (RIR fees, cheaper internet connections usually don't allow it) and to the internet as a whole (routing table size). Granted the costs of PI space are a bit higher with v4 than v6 because you have to actually purchase the addresses, but I suspect that is typically a minor factor. – Peter Green Jul 17 '20 at 16:12