How to reliably keep an SSH tunnel open?

Question

I use an SSH tunnel from work to go around various idotic firewalls (it's ok with my boss :)). The problem is, after a while the ssh connection usually hangs, and the tunnel is broken.

If I could at least monitor the tunnel automatically, I could restart the tunnel when it hangs, but I haven't even figured a way of doing that.

Bonus points for the one who can tell me how to prevent my ssh connection from hanging, of course!

Answer 1

Sounds like you need autossh. This will monitor an ssh tunnel and restart it as needed. We've used it for a couple of years and it seems to work well.

autossh -M 20000 -f -N your_public_server -R 1234:localhost:22 -C

More details on the -M parameter here

Answer 2

All stateful firewalls forget about a connection after not seeing a packet for that connection for some time (to prevent the state tables from becoming full of connections where both ends died without closing the connection). Most TCP implementations will send a keepalive packet after a long time without hearing from the other side (2 hours is a common value). If, however, there is a stateful firewall which forgets about the connection before the keepalive packets can be sent, a long-lived but idle connection will die.

If that is the case, the solution is to prevent the connection from becoming idle. OpenSSH has an option called ServerAliveInterval which can be used to prevent the connection from being idle for too long (as a bonus, it will detect when the peer died sooner even if the connection is idle).

Answer 3

On your own mac or linux machine configure your ssh keep the server ssh alive every 3 minutes. Open a terminal and go your your invisible .ssh in your home:

cd ~/.ssh/ 

then create a 1 line config file with:

echo "ServerAliveInterval 180" >> config

you should also add:

ServerAliveCountMax xxxx (high number)

the default is 3 so ServerAliveInterval 180 will stop sending after 9 minutes (3 of the 3-minute interval specified by ServerAliveInterval).

Answer 4

I've used the following Bash script to keep spawning new ssh tunnels when the previous one dies. Using a script is handy when you don't want or can't install additional packages or use compiler.

while true
do
  ssh <ssh_options> [user@]hostname
  sleep 15
done

Note that this requires a keyfile to establish the connection automatically but that is the case with autossh, too.

Answer 5

Systemd is ideally suited for this.

Create a service file /etc/systemd/system/sshtunnel.service containing:

[Unit]
Description=SSH Tunnel
After=network.target

[Service]
Restart=always
RestartSec=20
User=sshtunnel
ExecStart=/bin/ssh -NT -o ServerAliveInterval=60 -L 5900:localhost:5900 user@otherserver

[Install]
WantedBy=multi-user.target

(Modify the ssh command to suit)

• this will run as user sshtunnel so make sure that user exists first
• issue systemctl enable sshtunnel to set it to start at boot time
• issue systemctl start sshtunnel to start immediately

---

Update Jan 2018: some distros (e.g. Fedora 27) may use SELinux policy to prevent the use of SSH from systemd init, in which case a custom policy will need to be created to provide the necessary exemptions.

Answer 6

It sure looks to me that you're all misinterpreting ServerAliveCountMax. As I understand the docs, it is the number of server alive messages which can go unanswered without the connection being terminated. So in cases like we're discussing here, setting it to a high value will just ensure that a hung connection will not be detected and terminated!

Simply setting ServerAliveInterval should be sufficient to solve the problem with a firewall forgetting about the connection, and leaving ServerAliveCountMax low will allow the originating end to notice the failure and terminate if the connection fails anyway.

What you want is, 1) for the connection to stay open permanently under normal circumstances, 2) for connection failure to be detected and the originating side to exit on failure, and 3) for the ssh command to be re-issued every time it exits (how you do that is very platform dependent, the "while true" script suggested by Jawa is one way, on OS X I actually set up a launchd item).

Answer 7

Always use ServerAliveInterval SSH option in case the tunnel issues are generated by expired NAT sessions.

Always use a respawning method in case the connectivity goes down entirely, you have at least three options here:

• autossh program
• bash script (while true do ssh ...; sleep 5; done) do not remove the sleep command, ssh may fail quickly and you'll respawn too many processes

• /etc/inittab, to have access to a box shipped and installed in another country, behind NAT, without port forwarding to the box, you can configure it to create an ssh tunnel back to you:

  tun1:2345:respawn:/usr/bin/ssh -i /path/to/rsaKey -f -N -o "ServerAliveInterval 180" -R 55002:localhost:22 user@publicip 'sleep 365d'
  

• upstart script on Ubuntu, where /etc/inittab is not available:

  start on net-device-up IFACE=eth0
  stop on runlevel [01S6]
  respawn
  respawn limit 180 900
  exec ssh -i /path/to/rsaKey -N -o "ServerAliveInterval 180" -R 55002:localhost:22 user@publicip
  post-stop script
      sleep 5
  end script
  

or always use both methods.

Answer 8

I solved this problem with this:

Edit

~/.ssh/config

And add

ServerAliveInterval 15
ServerAliveCountMax 4

According to man page for ssh_config:

ServerAliveCountMax
         Sets the number of server alive messages (see below) which may be
         sent without ssh(1) receiving any messages back from the server.
         If this threshold is reached while server alive messages are
         being sent, ssh will disconnect from the server, terminating the
         session.  It is important to note that the use of server alive
         messages is very different from TCPKeepAlive (below).  The server
         alive messages are sent through the encrypted channel and there‐
         fore will not be spoofable.  The TCP keepalive option enabled by
         TCPKeepAlive is spoofable.  The server alive mechanism is valu‐
         able when the client or server depend on knowing when a connec‐
         tion has become inactive.

         The default value is 3.  If, for example, ServerAliveInterval
         (see below) is set to 15 and ServerAliveCountMax is left at the
         default, if the server becomes unresponsive, ssh will disconnect
         after approximately 45 seconds.  This option applies to protocol
         version 2 only.

 ServerAliveInterval
         Sets a timeout interval in seconds after which if no data has
         been received from the server, ssh(1) will send a message through
         the encrypted channel to request a response from the server.  The
         default is 0, indicating that these messages will not be sent to
         the server.  This option applies to protocol version 2 only.

Answer 9

ExitOnForwardFailure yes is a good adjunct to the other suggestions. If it connects but can't establish the port forwarding it's just as useless to you as if it hadn't connected at all.

Answer 10

I've had a need to maintain an SSH-tunnel long-term. My solution was running from a Linux server, and it's just a small C program that respawns ssh using key-based authentication.

I'm not sure about the hanging, but I've had tunnels die due to timeouts.

I would love to provide the code for the respawner, but I can't seem to find it right now.

Answer 11

while there are tools like autossh that helps to restart ssh session... what i find to be really useful is to run the 'screen' command. It allows you to RESUME your ssh sessions even after you disconnect. Especially useful if your connection is not as reliable as it should be.

• http://www.howtogeek.com/howto/ubuntu/keep-your-ssh-session-running-when-you-disconnect/

...don't forget to mark this is the 'correct' answer if it helps you k! ;-)

Answer 12

A bit of a hack, but I like to use screen to keep this. I currently have a remote forward that has been running for weeks.

Example, starting locally:

screen
ssh -R ......

When the remote forward is applied, and you have a shell on the remote computer:

screen
Ctrl + a + d

You now have an uninterrupted remote forward. The trick is to run screen on both ends

Answer 13

Recently had this issue myself, because these solutions require you to re-enter the password every time if you use a password login I used sshpass in a loop along with a text prompt to avoid having the password in the batch file.

Thought I'd share my solution on this thead in case anyone else has the same issue:

#!/bin/bash
read -s -p "Password: " pass
while true
do
    sshpass -p "$pass" ssh user@address -p port
    sleep 1
done

Answer 14

As autossh does not meet our needs (it exists with error if it can't connect to the server at the very first attempt), we've written a pure bash application: https://github.com/aktos-io/link-with-server

It creates a reverse tunnel for the NODE's sshd port (22) on the server by default. If you need to perform any other actions (like forwarding additional ports, sending mails on connection, etc...) you can place your scripts on-connect and on-disconnect folders.

Answer 15

You could very simply use tmux command on your remote server. It allows you to open a "session" on the remote machine where your will run your remote code.

Then you can exit (or lose the connection in your case) the remote machine without worrying given that your code is safely running in your tmux session.

Finally, when you reconnect on the remote server, you only need to tmux attach to come back to the session you had previously open and that's it.

Answer 16

I have had similar problems with my previous ISP. For me it was the same with any tcp connection, visiting websites or sending mail.

The solution was to configure a VPN connection over UDP(I was using OpenVPN). This connection was more tolerant to whatever caused the disconnections. Then you can run any service through this connection.

There can still be issues with the connection but since the tunnel will be more tolerant any ssh session will feel a short holdup rather than being disconnected.

To do this you will need a VPN service online which you can setup on your own server.