5

I manage an application that connects to various servers using a client-specific keypair. We have around 70 customers; all but one can connect to our FTP server (for SCP) after first accepting the RSA fingerprint (which hasn't changed in ~5 years).

To illustrate this behavior, I'm trying to SCP to the server every 15 minutes and dumping the result to /tmp/scp_(timestamp).log. The one client that can't connect was first presented with a round-robin of three different fingerprints:

$ grep -hA1 'The fingerprint for the RSA key' /tmp/scp_*.log | grep -E '[0-9a-f:]{16}' | sort | uniq -c
    129 14:4c:06:43:01:53:81:ef:b7:fd:09:46:91:06:c1:c9.
     98 34:9c:3f:13:f5:c4:74:9c:bd:b0:ff:4e:63:aa:eb:4c.
     97 36:dd:a7:90:4c:71:06:95:3c:e6:f3:ad:2a:96:c3:6a.

Recently, though, I started seeing truly random (to me, anyway) fingerprints:

$ grep -hA1 'The fingerprint for the RSA key' /tmp/scp_*.log | grep -E '[0-9a-f:]{16}' | sort | uniq -c | head
      1 02:5f:20:15:68:ea:0e:69:ef:7a:cc:1a:00:94:3f:96.
      1 02:a0:62:65:bc:41:6b:35:cf:4c:c2:fc:66:72:d8:5a.
      1 02:bb:5a:18:f1:ea:ca:71:f1:52:12:16:8c:85:5b:cc.
      3 02:c6:73:85:a9:94:82:f6:7e:51:a9:26:e7:d3:f7:7f.
      2 03:53:8c:74:b5:c1:dd:e4:7d:4b:17:e1:05:47:60:68.
      1 03:b3:c8:c6:c1:ef:54:28:65:4c:5f:73:f4:43:39:93.
      2 03:c7:8a:76:77:39:55:96:2e:c0:13:4a:21:f2:0d:51.
      2 06:8d:17:f9:cd:e0:10:4c:d0:44:58:8b:66:f8:f5:a8.
      1 07:fa:59:2b:90:96:4a:4c:85:eb:4a:37:91:d7:8e:0f.
      2 0d:21:b5:86:8e:ae:4e:97:87:f6:42:c7:e4:11:c0:4a.
$ grep -hA1 'The fingerprint for the RSA key' /tmp/scp_*.log | grep -E '[0-9a-f:]{16}' | sort | uniq -c | wc -l
253

I was confused by only three fingerprints but at this point I'm baffled by this number (253!!!) and ready to throw in the towel on trying to understand why this is happening. All I really have been able to figure out is that this started mid-November 2019.

But wait, there's more. Somehow, after seeing no successful authentications after this mysterious event in November, I saw some successful connections last week!

$ grep '^2019.*abc.*authenticated' sftp_8{1,2}.log | tail -2
2019-11-17 13:00:02,623 [18508]: user 'abc' authenticated via 'publickey' method
2019-11-17 13:00:17,351 [18514]: user 'abc' authenticated via 'publickey' method
$ grep '^2020.*abc.*authenticated' sftp_8{1,2}.log
2020-01-03 16:13:01,874 [6339]: user 'abc' authenticated via 'publickey' method
2020-01-03 16:17:21,147 [6665]: user 'abc' authenticated via 'publickey' method
2020-01-03 16:18:46,564 [6713]: user 'abc' authenticated via 'publickey' method
2020-01-03 16:19:02,504 [6825]: user 'abc' authenticated via 'publickey' method
2020-01-03 16:15:10,228 [24179]: user 'abc' authenticated via 'publickey' method
2020-01-03 16:18:57,216 [24494]: user 'abc' authenticated via 'publickey' method
$

To further muddy the waters, the customer server is not our hardware so I'm contractually limited in what I can do on this server.

Again, N-1 out of N (~70) customers can connect with the fingerprint I've seen for five years. I'm just looking for a push in the right direction. Thanks for reading!

Andrew
  • 51
  • 1
  • 3
    One thing I've done to foil the kind of attack this smells like is [using a SSH CA to sign host keys](https://security.stackexchange.com/a/69225/56587) (and a second one to sign client keys). And to be clear, a customer I can't name *really has* been subject to a transparent man-in-the-middle on SSH connections outbound from their facility, so the attack is not merely theoretical. – Charles Duffy Jan 08 '20 at 21:28
  • 2
    ...that said, in the case where I saw it, the attacker was smart enough to cache their locally-generated keys so systems using TOFU (where that "first use" was after they put their infrastructure in place) wouldn't catch it. Generating a new key each time is sloppy; going to get caught that way when users investigate why they're getting a warning from SSH about unknown host keys. – Charles Duffy Jan 08 '20 at 21:29

0 Answers0