41

This is a simple issue that we all face and probably resolve manually without giving much thought.

As servers change, are re-provisioned, or IP addresses reallocated, we receive the SSH host verification message below. I'm interested in streamlining the workflow to resolve these ssh identification errors.

Given the following message, I typically vi /root/.ssh/known_hosts +434 and remove (dd) the offending line.

I've seen developers/users in other organizations delete their entire known_hosts file out of frustration in seeing this message. While I don't go that far, I know there's a more elegant way to handle this.

Tips?

[root@xt ~]# ssh las-db1

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
ed:86:a2:c4:cd:9b:c5:7a:b1:2b:cc:42:15:76:8c:56.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:434
RSA host key for las-db1 has changed and you have requested strict checking.
Host key verification failed.
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • 1
    Why not copy the old key over when reinstalling / reallocating a server? – Random832 Aug 13 '12 at 13:11
  • @Random832 It does not scale in a situation where you have a lot of servers... Or if you have a lab/test environment where you're recycling IP addresses often. Or another case; an unplanned server replacement where the original server is unavailable... – ewwhite Aug 13 '12 at 13:14
  • 3
    The big question i have is: How do you get into this situation? If you're re-using hostnames, you might want to reconsider your host naming standard (http://tools.ietf.org/html/rfc1178). If you're re-using IP addresses, you should have the decommissioning of ssh keys as part of the same procedure as decommissioning the previous server on that IP address. If you're working in a "webscale" environment where IP reuse happens on an automated basis and semi-meaningless hostnames are mandatory, then you should have a well-automated enough server lifecycle process that fixing the ssh keys will procee – Paul Gear Aug 16 '12 at 01:07
  • We have the same problem. I don't have experience with NASA's Mesh http://ti.arc.nasa.gov/opensource/projects/mesh/ but it is on my list of technologies to review. – Andrew Gilmartin Aug 13 '12 at 00:24

5 Answers5

46

You can use the ssh-keygen command to remove specific entries by host:

ssh-keygen -R las-db1

If you don't have that command, you could always use sed:

sed -i '/las-db1/d' /root/.ssh/known_hosts
Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
  • 3
    Using ssh-keygen -R to resolve the problem with conflicting keys is also what the ssh client in Ubuntu suggests in the error message when this problem occurs. – Kenny Rasschaert Aug 12 '12 at 21:03
  • I was not aware of that switch to the `ssh-keygen` command. Good! – ewwhite Aug 12 '12 at 21:07
  • 10
    Remember to check and be sure that someone isn't actually "DOING SOMETHING NASTY!" – Michael Hampton Aug 12 '12 at 21:21
  • 1
    Make sure you don't do this at a conference! – Paul Gear Aug 14 '12 at 23:30
  • And as a side, once you've run `ssh-keygen -R las-db1`, how do you eliminate/remove the first-time `Are you sure you want to continue connecting` message? – ewwhite Aug 15 '12 at 18:38
  • 2
    @ewwhite You populate the `/etc/ssh/known_hosts` file on your network with appropriate host keys (managed w/ puppet, etc.), or you populate your personal ~/.ssh/known_hosts file (to do either of these you can run `ssh-keyscan` over a known good/secure connection). Barring that (and assuming you care not at all about security) set `UserKnownHostsFile=/dev/null` and `StrictHostKeyChecking=no` in your `~/.ssh/config file` (or pass the options to ssh with `-o`) – voretaq7 Aug 15 '12 at 18:44
  • Remember that if you uses hostname along with IP address you must remove both of them, if don't, sometimes the message to accept new key continue appearing. – Sotsir Sep 28 '15 at 17:06
25

As a puppet user, my method for resolving this is to actually have my puppet server collect my SSH host keys and publish them to all my systems that make SSH connections.

This way I don't have to worry about removing them. Ninety-nine percent of the time puppet has run and updated the keys for me since I have my agents running every thirty minutes. The exceptions for me are very rare, and so I don't mind a quick edit of the system wide known_hosts if I am not willing to wait.

class ssh::hostkeys {

  @@sshkey { "${::clientcert}_rsa":
    type => rsa,
    key => $sshrsakey,
    tag => 'rsa_key',
  }

  if 'true' == $common::params::sshclient {
    Sshkey <<| tag == 'rsa_key' |>> {
      ensure => present
    }
  }

  file {'/etc/ssh/ssh_known_hosts':
    ensure => present,
    owner => 'root',
    group => 'root',
    mode => 0644,
  }

}
Zoredache
  • 128,755
  • 40
  • 271
  • 413
4

I would like to add a suggestion that can help you in very specific cases where security is of less concern.

I have a lab environment with machines that get reinstalled often. Every time that happens, new host keys get generated (I could probably save the host key somewhere and set it in the post-install script).

Since security is not an issue for me in this lab environment, and the keys change so often, I have the following in my .ssh/config file:

Host lab-*
  User kenny
  IdentityFile ~/.ssh/lab_id_rsa
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null

This makes sure that connecting to my lab machines will never cause that error again and my ssh client will just connect without checking the host key.

This is something you should only do if security is of no concern to you at all, because this puts you in a vulnerable position for a man-in-the-middle attack.

Kenny Rasschaert
  • 8,925
  • 3
  • 41
  • 58
  • 1
    Seems risky. I'd like to keep the host verification since some of these are public-facing systems. – ewwhite Aug 12 '12 at 22:27
  • 1
    This will be why he mentioned that this method is only for "where security is of less/no concern". Private networks/lab environments are perfect for this configuration. Multi-machine auto-scaling production/public facing systems, not so much. – dannosaur Oct 02 '14 at 17:06
4

According to openssh 5.6 release note , you don't need to use hosts keys anymore :

Hostbased authentication may now use certificate host keys. CA keys must be specified in a known_hosts file using the @cert-authority marker as described in sshd(8).

Warning : I've never heard of anyone using this feature in production so far. Use at your own risk (but I believe openssh developpers are paranoid enough not to release such a killer feature without a lot of testing and code auditing).

2

SSHFP DNS ResourceRecord may help depending on how much your ssh client takes advantage of it. Store all your ssh public key fingerprints up there with the host name.

Implement DNSSEC or DNS over SSL beforehand.
http://www.ietf.org/rfc/rfc4255.txt

FreeIPA.org handles host and user key management as well as PKI Certificates. It also automatically uploads SSHFP DNS records when new keys are created.

The System Security Services Daemon (SSSD) can be configured to cache and retrieve host SSH keys so that applications and services only have to look in one location for host keys. Because SSSD can use FreeIPA as one of its identity information providers, FreeIPA provides a universal and centralized repository of keys. Administrators do not need to worry about distributing, updating, or verifying host SSH keys.

http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/host-keys.html

rjt
  • 568
  • 5
  • 25