Differing SSH known_hosts formats

Question

I've been having issues with my CI server's deployment lately due to the client (CI) rejecting the remote's host key (despite it being present in known_hosts). I was stumped until today, when I realized that SSH was saving host keys in a format that the deployment plugin doesn't seem to be compatible with. For reference, the compatible format (still present on my personal machine) resembles this:

11.22.33.44 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkVf7rhfC7nLxbeIQRj2bWitUC+XLSAeQ0ap8r8rKObDXYfPdB97NZth9JCEt3OrBXuBeg4PaAEuPu2QF7WXoT60hgAP6etr0W4LqcH59yd/X0ogFP7Y7hIf6dz1txDKaW92wgUi5XShwH6vukf0gLvW6/ak1LTBuoy72gaoUvxZge4KZivz9XqvSQHNOG9KYNfh8U6cRM8YTQo5in7YD5d6REV/FUmXpvBzCa9kbVRSlQFGYEc1HidTnPnJDteas3A9y3na385O7WN64aAkg7TO8IFXKdDHSwji9ZyrCVPA5GEuyLKhDFanV8iJ7CNflHMP8TwG5FOT2bSkV0lPyl

While the format SSH is currently saving when accepting new host keys resembles this:

11.22.33.44 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJEs165NgdEcD94Xg3ySFA/qgkfytxNCX1X3pB2SPgU/mHLGXCXM8+VqMBXocM8OMOq2L0fDGr5mI+nGqjhNU=

(Note: while I fudged the public keys a bit, they still don't resemble each other at all in their original form.)

Only the first format is compatible with the deployment plugin, while the second is unconditionally ignored. Can anyone explain this discrepancy?

Answer 1

These are not different formats of known_hosts, but different key types (ssh-rsa and ecdsa-sha2-nistp256 - well described in manual page for sshd). Server usually have more host keys of different types to provide wider compatibility with different clients.

If you are on the server, you can find all the host keys and print their public keys using, but the line is not in the same format as :

$ cat /etc/ssh/ssh_host_*.pub
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJEs165NgdEcD94Xg3ySFA/qgkfytxNCX1X3pB2SPgU/mHLGXCXM8+VqMBXocM8OMOq2L0fDGr5mI+nGqjhNU= user@host

The format that is accepted by known_hosts file can be obtained using (from the server to achieve the authenticity of the keys):

$ ssh-keyscan 11.22.33.44
11.22.33.44 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJEs165NgdEcD94Xg3ySFA/qgkfytxNCX1X3pB2SPgU/mHLGXCXM8+VqMBXocM8OMOq2L0fDGr5mI+nGqjhNU=
#[...]

This prints the format you can directly store in the client known_hosts.

For the whole picture (from manual page):

Each line in these files contains the following fields: markers (optional), hostnames, bits, exponent, modulus, comment. The fields are separated by spaces.

(though it looks non-consistent with what is generated: hostname, key-type, key data (base64)) - I will check that later, since it is not important for the question