4

While not intending to cross-post, after sending this question to the OpenSSH list on SecurityFocus, I noticed that the list is rather low traffic (the previous post was approximately 5 months prior). That being said, I decided to repost here as the issue will probably get more eyeballs (and will stand a better chance of being of use to others if answered):

The issue: I have a reverse SSH tunnel from an internal machine to a host in my DMZ which is set to launch at system boot and relaunch if the tunnel fails. However, when the tunnel is interrupted (for example, due to a network outage), it cannot reestablish due to the port on the DMZ host being in use. From my read of the OpenSSH mailing list archives and elsewhere, this appears to be because the port is in a TIME_WAIT state. This is fine: I can put in a sleep statement in the script that sets up the tunnel. However, this leads to two questions:

1) Any idea how to determine what the TIME_WAIT interval is defined as on a particular Linux (or other) system? While I could just sleep for 5 minutes and be fine, it would be good to shave off as much time as possible. 2) While OpenSSH doesn't appear to support the "ClearAllForwardings" option, is there similar functionality whereby an auth'd connection can automatically teardown and recreate an existing connection it had previously established?

A long sleep would probably be "good enough" but I'd prefer to handle the TIME_WAIT condition in a more efficient manner if possible.

I appreciate any guidance or suggestions!
mjbraun
  • 139
  • 8
  • The first thing to do is to check whether the socket really is sitting there in TIME_WAIT - as far as I can see from your description this is currently just an inference rather than something you've actually checked directly. – Richard Kettlewell May 30 '12 at 07:50

2 Answers2

3

I would think you could play with various SSH settings such as TCPKeepAlive, ServerAliveInterval, ServerAliveCountMax, etc to set up where if the connection goes down it will kill everything. I have a similar setup and I have made a lot of modifications to both SSHD and SSH on both sides to match up with what I want. Then I have a cron job that runs every 5 min that restarts the tunnel if I need to.

#!/bin/bash
if ps aux | grep "ssh -fnNTx" | grep -v "grep"
then
echo "Already Running"
else
echo "Starting now"
ssh -fnNTx -L 1514:127.0.0.1:514 user1@X.X.X.X
fi

So far this solution has worked fine for me. You can also set up some type of Nagios check or another script to see if the tunnel is open and if not, do a kill of that pid so it can restart.

Edit:

Previous article that talks a lot about TIME_WAIT issues. How to forcibly close a socket in TIME_WAIT?

Community
  • 1
Eric
  • 1,373
  • 3
  • 17
  • 33
  • Eric, thank you for your response! I'm already using the TCPKeepAlive and ServerAlive intervals on the client end, and my script correctly attempts to respawn the tunnel in case of failure. The problem is on the server side: until the port that's in a TIME_WAIT state closes, the client cannot bind to it again. I'll probably go the "sleep" route, as this seems to be something outside of SSHD's control. – mjbraun May 29 '12 at 17:27
  • 1
    Please read this post. It has a lot of information and some setting files about TIME_WAIT. Espcially Leigh Caldwell's post. http://serverfault.com/questions/329845/how-to-forcibly-close-a-socket-in-time-wait – Eric May 29 '12 at 17:34
1

SSHD should set SO_REUSEADDR, allowing the new instance to bind even if the previous instance still has connections in TIME_WAIT state. Either you have a buggy SSHD or you have some configuration setting that inhibits this behavior (for example, if you disabled X11UseLocalHost).

David Schwartz
  • 31,215
  • 2
  • 53
  • 82