47

I've grown quite fond of HTTP reverse proxies in our development environment and found the DNS based virtual host reverse proxy quite useful. Having only one port (and the standard one) open on the firewall makes it much easier for management.

I'd like to find something similar to do SSH connections but haven't had much luck. I'd prefer not to simply use SSH tunneling since that requires opening port ranges other than the standard. Is there anything out there that can do this?

Could HAProxy do this?

splattne
  • 28,348
  • 19
  • 97
  • 147
ahanson
  • 1,674
  • 2
  • 15
  • 21
  • Is your intended application file transfer, or actual SSH access to hosts? – Kyle Jul 01 '09 at 16:20
  • Direct SSH access to the host is the goal. This need comes from the desire to run a Mercurial server in-house but our server is behind the firewall. Right now I'm working on simply setting up an HTTP version but I wanted to have commits use SSH instead of HTTP. Direct SSH access to other servers would be a bonus if this was possible. – ahanson Jul 01 '09 at 19:01
  • This is such an annoying problem. – bias Dec 22 '11 at 18:36
  • My understanding is that HAProxy now supports this via SNI. Although I have not been able to get this going yet. – Carel Jan 07 '18 at 12:30

7 Answers7

26

I don't believe name-based SSH is something that will be possible given how the protocol works.

Here are some alternatives.

  • You could do is setup the host that answers for port 22 to act as a gateway. Then you can configure the ssh server to forward requests to the inside based on the key. SSH Gateway example with keys

  • You could adjust your client to use that host as a proxy. That is, it would ssh to the gateway host, and then make use that host to make a connection to the internal host. SSH proxy with client configuration.

  • You could also setup a simple http proxy on the edge. Then use that to allow incoming connections. SSH via HTTP proxy.

Obviously with all the above, making sure you properly configure and lock down the gateway is pretty important.

Zoredache
  • 128,755
  • 40
  • 271
  • 413
25

I have been searching for a solution for this problem on and off for the last 16 months. But each time I look, it seems impossible to do this with the SSH protocol as specified in relevant RFCs and implemented by major implementations.

However if you are willing to use a slightly modified SSH client and you are willing to utilize protocols in way that was not exactly intended when they were designed, then it is possible to achieve. More on this below.

Why it is not possible

The client does not send the hostname as part of the SSH protocol.

It might send the hostname as part of a DNS lookup, but that might be cached, and the path from client through resolvers to authoritative servers might not cross the proxy, and even if it did there is no robust way of associating specific DNS lookups with specific SSH clients.

There is nothing fancy you can do with the SSH protocol itself either. You have to pick a server without even having seen the SSH version banner from the client. You have to send a banner to the client, before it will send anything to the proxy. The banners from the servers could be different, and you have no chance of guessing, which one is the correct one to use.

Even though this banner is sent unencrypted, you cannot modify it. Every bit of that banner will be verified during connection setup, so you'd be causing a connection failure a bit down the line.

The conclusion to me is pretty clear, one has to change something on the client side in order to make this connectivity work.

Most of the workarounds are encapsulating the SSH traffic inside a different protocol. One could also imagine an addition to the SSH protocol itself, in which the version banner send by the client include the hostname. This can remain compatible with existing severs, since part of the banner is currently specified as a free form identification field, and though clients typically wait for the version banner from the server before sending their own, the protocol does permit the client to send their banner first. Some recent versions of the ssh client (for example the one on Ubuntu 14.04) does send the banner without waiting for the server banner.

I don't know of any client, which has taken steps to include the hostname of the server in this banner. I have send a patch to the OpenSSH mailing list to add such a feature. But it was rejected based on a desire to not reveal the hostname to anybody snooping on the SSH traffic. Since a secret hostname is fundamentally incompatible with the operation of a name based proxy, don't expect to see an official SNI extension for the SSH protocol anytime soon.

A real solution

The solution that worked best for me was actually to use IPv6.

With IPv6 I can have a separate IP address assigned to each server, so the gateway can use the destination IP address to find out which server to send the packet to. The SSH clients might sometimes be running on networks where the only way to get an IPv6 address would be by using Teredo. Teredo is known to be unreliable, but only when the native IPv6 end of the connection is using a public Teredo relay. One can simply put a Teredo relay on the gateway, where you'd run the proxy. Miredo can be installed and configured as a relay in less than five minutes.

A workaround

You can use a jump host/bastion host. This approach is intended for cases where you don't want to expose the SSH port of the individual servers directly to the public internet. It does have the added benefit on reducing the number of externally facing IP address you need for SSH, which is why it is usable in this scenario. The fact that it is a solution intended to add another layer of protection for security reasons doesn't prevent you from using it when you don't need the added security.

ssh -o ProxyCommand='ssh -W %h:%p user1@bastion' user2@target

A dirty to hack to make it work if the real solution (IPv6) is outside of your reach

The hack I am about to describe should only be used as the absolutely last resort. Before you even think about using this hack, I strongly recommend getting an IPv6 address for each of the servers which you want to be externally reachable through SSH. Use IPv6 as your primary method for accessing your SSH servers and only use this hack when you need to run an SSH client from an IPv4-only network where you don't have any influence on getting IPv6 deployed.

The idea is that traffic between client and server need to be perfectly valid SSH traffic. But the proxy only need to understand enough about the stream of packets to identify the hostname. Since SSH doesn't define a way to send a hostname, you can instead consider other protocols which do provide such a possibility.

HTTP and HTTPS both allow for the client to send a hostname before the server has send any data. The question now is, whether it is possible to construct a stream of bytes which is simultaneously valid as SSH traffic and as either HTTP and HTTPS. HTTPs it is pretty much a non-starter, but HTTP is possible (for sufficiently liberal definitions of HTTP).

SSH-2.0-OpenSSH_6.6.1 / HTTP/1.1
$: 
Host: example.com

Does this look like SSH or HTTP to you? It is SSH and completely RFC compliant (except some of the binary characters got mangled a bit by the SF rendering).

The SSH version string includes a comment field, which in the above has the value / HTTP/1.1. After the newline SSH has some binary packet data. The first packet is a MSG_SSH_IGNORE message send by the client and ignored by the server. The payload to be ignored is:

: 
Host: example.com

If an HTTP proxy is sufficiently liberal in what it accepts, then the same sequence of bytes would be interpreted as an HTTP method called SSH-2.0-OpenSSH_6.6.1 and the binary data at the start of the ignore message would be interpreted as an HTTP header name.

The proxy would understand neither the HTTP method or the first header. But it could understand the Host header, which is all it needs in order to find the backend.

In order for this to work the proxy would have to be designed on the principle that it only needs to understand enough HTTP to find the backend, and once the backend has been found the proxy will simply pass the raw byte stream and leave the real termination of the HTTP connection to be done by the backend.

It may sound like a bit of a stretch to make so many assumptions about the HTTP proxy. But if you were willing to install a new piece of software developed with the intention to support SSH, then the requirements for the HTTP proxy don't sound too bad.

In my own case I found this method to work on an already installed proxy with no changes to code, configuration, or otherwise. And this was for a proxy written with only HTTP and no SSH in mind.

Proof of concept client and proxy. (Disclaimer that proxy is a service operated by me. Feel free to replace the link once any other proxy has been confirmed to support this usage.)

Caveats of this hack

  • Don't use it. It's better to use the real solution, which is IPv6.
  • If the proxy attempts to understand the HTTP traffic, it is surely going to break.
  • Relying on a modified SSH client isn't nice.
kasperd
  • 29,894
  • 16
  • 72
  • 122
  • Can you elaborate what you mean by `the gateway can use the destination IP address to find out which server to send the packet to`, in the IPv6 solution? – Jacob Ford May 24 '18 at 19:56
  • @JacobFord The key to understanding that sentence is that I use the word **gateway** not **proxy**. With IPv6 you get enough IP addresses to assign one to every host and still have addresses to spare. All of the machines you'd put behind the proxy would have their own IP address which is what you'd have the names resolve to. So you no longer need a proxy, the clients send their traffic to the IP address of the desired host, and you can route the traffic directly there. – kasperd May 24 '18 at 21:40
  • for the workaround part, there is a relatively new command switch for ssh `-J`, so you can use: ssh -J user1@front user2@target – PMN Oct 28 '19 at 14:51
  • 1
    @kasperd In the case of IPv6, all the hosts will have a IPV6 and client connects to it directly right? But won't that make the hosts publicly accessible on the internet? Let us say company policy allows to have only private IP addresses (in the case of AWS) how will this solution works? – Navaneeth K N Dec 06 '19 at 04:01
  • Wow! Such a thorough well-rounded answer. Thank you @kasperd :) – Sam Sirry May 11 '22 at 03:27
  • Can I copy your proxy service? It's not for commercial use. I share your idea that it's for fixing the problem of slow IPv6 implementation. – Sam Sirry May 11 '22 at 03:29
4

There is a new kid on the block. SSH piper will route your connection based on pre-defined usernames, that should be mapped to the inner hosts. This is the best solution in the context of of reverse-proxy.

SSh piper on github

My first tests confirmed, SSH and SFTP both work.

Király István
  • 327
  • 2
  • 10
  • That is effectively a MITM-attack. I expect it to break in every setup that is protected against MITM-attacks. – kasperd Oct 06 '17 at 20:38
  • 1
    Actually the hosting party is both, the host and the man in the middle, thus only those two are involved. – Király István Oct 09 '17 at 23:35
  • @KirályIstván this is awesome project and proofs that other answers are not 100% correct. I created similar tool based on based on https://github.com/gliderlabs/sshfront and some bash/python (I can't open source it but it's not that interesting - bash runs ssh client and python is used as auth handler). But I have one issue with current solution. It dosen't support sftp (scp works). It hangs trying to run subsystem `debug1: Sending subsystem: sftp` . It turns out that shirt-front doesn't support subsystems. Do you support hem in SSh piper? – Maciek Sawicki Mar 27 '18 at 12:26
  • 1
    @MaciekSawicki I'm not the author. I just use it in my project. .) – Király István Mar 29 '18 at 00:16
3

I don't believe this is something that would be possible, at least the way you described, although I would love to be proved wrong. It doesn't appear that the client sends the hostname that it wishes to connect to (at least in the clear). The first step of the SSH connection seems to be to set up encryption.

In addition, you would have issues with host key verification. SSH clients will verify keys based on an IP address as well as a hostname. You would have multiple hostnames with different keys, but the same IP you're connecting to.

A possible solution would be to have a 'bastion' host, where clients can ssh in to that machine, get a normal (or restricted if desired) shell, and can then ssh into internal hosts from there.

Mark
  • 2,846
  • 19
  • 13
  • The bastion concept is what we current have setup but is problematic for my version control (without extra effort). – ahanson Jul 01 '09 at 19:05
  • It's super annoying that ssh doesn't send the fqdn and handles the DNS on the client side. I guess people didn't worry about public IP bloat and NAT when most TCP/IP application level protocols were created. Because, seriously, you **should** be able to do FQDN based NAT with iptables (i.e. kernel filters). – bias Dec 22 '11 at 18:41
  • The hostkey could be the key of the reverse proxy. This is a benefit if you trust the security of backend, since you have control of the internal network and hosts. – rox0r Jun 18 '13 at 21:52
  • @bias You can't do NAT based on hostname, even if the higher level protocol does send the hostname. The reason it cannot be done is, that the choice of backend has to be done when the SYN packet is received, but the hostname is only sent after the SYN has been processed and a SYN-ACK has been returned. You can however use a very thin application layer proxy for protocols that do send hostname, such as http and https. – kasperd Jul 08 '14 at 13:57
2

I am wondering if Honeytrap (the low interaction honeypot) that has a proxy mode couldn't be modified to achieve that.

This honeypot is able to forward any protocol to another computer. Adding a name based vhost system (as implemented by Apache) could make it a perfect reverse proxy for any protocol no ?

I do not have the skills to achieve that but maybe it could be a great project.

Kafeine
  • 29
  • 1
  • excellent idea! – bias Dec 22 '11 at 18:27
  • 2
    The vhost feature in Apache relies on the client sending the hostname before any data has been sent from the server. The SSH protocol doesn't involve such a hostname, so I don't see how you'd even start implementing such a feature. But if you can elaborate on your ideas, I'd be happy to implement a proof of concept or tell you, why it wouldn't work. – kasperd Jul 08 '14 at 15:01
1

because of how ssh works I think it's not possible. Similar to https you would have to have different (external) IPs for different hosts, because the gateway doesn't know where do you want to connect to because everything in ssh is encryted

Jure1873
  • 3,692
  • 1
  • 21
  • 28
  • I'm not sure if this is still correct. TLS now has something called SNI, or Server Name Indication. Basically, a part of the TLS message includes the host name, so you can read from it and redirect however you like. I don't believe Nginx has support for it, although HAProxy certainly does. – 157 239n Apr 04 '22 at 23:39
  • I think this might be what we are looking for: https://www.haproxy.com/blog/route-ssh-connections-with-haproxy/ Free for non-commercial use. – Jay M Jun 01 '22 at 18:53
0

You can use HAProxy on the server side and ssh + openssl on the client side to SSH over HTTPS. This approach uses HTTPS SNI to identify the destination server.

Example client command:

ssh -o ProxyCommand="openssl s_client -quiet -connect <proxy IP>:2222 -servername server1" dummyName1

dummyName1 in the above command is used purely as a placeholder since ssh requires a hostname, but openssl is doing all of the connection work here

Basic HAProxy config:

frontend fe_ssh
   bind *:2222 ssl crt /etc/haproxy/certs/ssl.pem
   mode tcp
   log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq dst:%[var(sess.dst)] "
   tcp-request content set-var(sess.dst) ssl_fc_sni
   use_backend %[ssl_fc_sni]

backend server1
   mode tcp
   server s1 <backend ip 1>:22 check

backend server2
   mode tcp
   server s2 <backend ip 2>:22 check

backend server3
   mode tcp
   server s2 <backend ip 3>:22 check

More details and security improvements can be found at the Source

John_K
  • 1
  • 1