11

I regularly connect via SSH to a remote server, from an Ubuntu system, on the default port 22. Let's call the server example.org. I am sure that this server is configured properly, and I have confirmed that the following issue is independent from my OS and persists across re-installs.

There is one particular Wifi access point where, if I connect to the server (ssh example.org), I get this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:[REDACTED].
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/user/.ssh/known_hosts:3
  remove with:
  ssh-keygen -f "/home/user/.ssh/known_hosts" -R "serv.org"
ED25519 host key for serv.org has changed and you have requested strict checking.
Host key verification failed.

The problematic access point belongs to an academic institution, and seems to be more locked-down than commercial ISP networks (for example I can't download torrents on it). If I go back to another network (say, using my phone as an access point), I can connect again.

According to Wireshark:

  • The DNS query (to 8.8.8.8) for example.org returns the same IP address, even on the problematic access point.
  • The SSH key exchange seems to happen as usual, but the key sent by the server in the "ECDH Key Exchange Reply" indeed has a different fingerprint when I am connecting through the problematic AP.

I don't understand what this network is doing. Blocking port 22 would be one thing, but here I seem to reach the server and get a wrong key as a response.

Could this access point be intentionally tampering with the SSH connection? Is there a way for me to securely use SSH over it despite this? Should I just avoid using it?

Hey
  • 214
  • 1
  • 10
  • 1
    Just FYI: if you need to use an example domain you should really use `example.com` or `example.org`. They are the **official** examples domains. If you use a random domain name it could lead people to actually end up on the site, or maybe the site could receive unwanted attention from your "publicity". – Giacomo Alzetta Nov 18 '19 at 17:42
  • @GiacomoAlzetta I didn’t know that, thank you! – Hey Nov 18 '19 at 18:23

3 Answers3

22

Either this systems host keys are changing or someone/something is MITM'ing the SSH connection.

The appropriate course of action is to consider that host as compromised (although its likely not the host itself, rather the connection) unless/until you have an explanation.

You may want to reach out to the system administrator of that AP and advise them of your concerns and try and track this down with them.

Joseph Sible-Reinstate Monica
  • 336
  • 2
  • 14
davidgo
  • 5,964
  • 2
  • 21
  • 38
19

Since it's an academic institution, I'm thinking it's probably a firewall/security gateway that intercepts and analyzes the SSH traffic for things like Data Loss Prevention (DLP) or general audit logging.

For example of such a firewall: https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/decryption/decryption-concepts/ssh-proxy.html

Rob Olmos
  • 2,220
  • 1
  • 15
  • 25
  • 5
    That’s probably what is happening here. On a side note, changing the ssh port is often enough to no get mitm by the so-called firewall. A quick check can be done trying to access github on 443 (intended for over restrictive corporate firewall) `ssh -T -p443 git@ssh.github.com` – Cyrbil Nov 17 '19 at 10:10
  • I can see this happening on a corporate network, but DLP on a student Wi-Fi?... – user1686 Nov 18 '19 at 12:27
  • @Cyrbil Good idea on changing the port. Interesting they would only use simple port-based interception and not do DPI and block SSH not using the standard port (if that's possible). – Rob Olmos Nov 18 '19 at 20:52
  • @grawity It's odd to me it's being done at all and it's bypassed by just changing the port. Maybe it's MITM malware (soft or hard). – Rob Olmos Nov 18 '19 at 20:58
4

The answer by davidgo is correct. Note that, as long as you're using pubkey auth and not password, this MITM does not compromise the safety of your credentials. But it does compromise the privacy and authenticity of anything transmitted over the session. Note that "authenticity of anything transmitted over the session" includes (rather, probably consists entirely of) commands sent to the shell or whatever process you're interacting with and output sent back.

I had originally suggested the following, which is dangerously wrong, at least without further measures:

One easy workaround is double-ssh. Assuming the host you're logging into allows port forwarding, login once with the compromised connection, using pubkey auth and with forwarding enabled to forward a local port to localhost:22, e.g. -L 12345:localhost:22. Now, you can run a second ssh session tunneled through the first, and unless the attacker is actively anticipating you doing this, this second session will have the right host fingerprint and you know that it's actually secure, not subject to interception by the MITM.

This was written hastily as a simplification from the right way I had in mind to do it, which is to have a separate login account used purely for forwarding, with no shell, which I believe is still safe to use with a compromised (MITM'd) connection, but I'd still advise caution. It may also be possible to setup your authorized_keys file to limit the key you use for the initial (MITM'd) login so that it's unable to issue any commands, only forward connections; if so, such a setup should be safe.

R.. GitHub STOP HELPING ICE
  • 707
  • 4
  • 14
  • 1
    If it compromises the authenticity of anything sent over the session, then even with your double-ssh trick, couldn't the attacker take over your outer session and take control of your target box through it? – Joseph Sible-Reinstate Monica Nov 18 '19 at 03:50
  • @JosephSible it's more security by obscurity, but it still should work as most institutions won't be actively blocking such niche tricks. – vikarjramun Nov 18 '19 at 05:17
  • 1
    @vikarjramun It's not just about blocking your connection. I mean wouldn't it let an attacker completely own your target box? – Joseph Sible-Reinstate Monica Nov 18 '19 at 05:20
  • @JosephSible I mean it would do the same thing even if you didn't attempt this second loop connection. However, this is a school, so they likely wouldn't try to take over the box but instead snoop on traffic. – vikarjramun Nov 18 '19 at 05:29
  • 1
    @vikarjramun that's my point. Just accepting the bad host key at all seems like a bad idea. And you're not sure whether it's "just" the school or not. – Joseph Sible-Reinstate Monica Nov 18 '19 at 05:55
  • 1
    @JosephSible I guess the idea is to open a broken outer connection, accepting (once) the MITM key, then pipe a secure connection with strict host key verification through it? If strict host key checking was disabled for the outer connection, but enabled for the inner one (and the correct host key is already in our `known_hosts`, as in my case), then the inner connection will either be secure or fail. Unless I missed something. – Hey Nov 18 '19 at 10:10
  • 2
    @Hey Maybe the inner connection succeeds and is secure, but the MITM still uses your outer connection to install a keylogger. Is that a success, or a failure? – user253751 Nov 18 '19 at 10:43
  • @user253751 Thanks; that's exactly the kind of attack I was trying to describe. – Joseph Sible-Reinstate Monica Nov 18 '19 at 12:38
  • 1
    Indeed this is dangerously wrong. Will fix. Thanks – R.. GitHub STOP HELPING ICE Nov 18 '19 at 15:20
  • @JosephSible: Please let me know if you think the answer is still bad. – R.. GitHub STOP HELPING ICE Nov 18 '19 at 15:24
  • @R.. Looks good now. – Joseph Sible-Reinstate Monica Nov 18 '19 at 16:12
  • @user253751: They can install all the keylogger they want on account without any writable directories and a shell that doesn't support <, >, or | – joshudson Jun 09 '22 at 15:53
  • @joshudson run `vi`, run the command `:!/bin/sh`, now you have a shell with redirection commands – user253751 Jun 09 '22 at 16:06
  • @user253751: I build this construction once for another purpose. /bin/sh yields the same crippled shell again. – joshudson Jun 09 '22 at 16:29
  • @joshudson then I just run `tee` to write files. Still have nowhere to write them to, but maybe there are some interesting device files around that I can write to directly. This is why people say not to bother with blacklisting. If there's nothing to write to, then you may as well allow <>| – user253751 Jun 09 '22 at 16:31