ssh: Error loading key "./id_rsa": invalid format

Question

For some reason one of my ssh keys "just broke" - it just stopped working:

$ ssh-add ./id_rsa
Error loading key "./id_rsa": invalid format

Copying the key inside a clean VM, the key does work. Even with the exact same ssh version (OpenSSH_7.8p1, OpenSSL 1.1.0i-fips 14 Aug 2018 on Fedora 28). So it must be related to some config on my system I assume.

# cat ./id_rsa
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,...

...
-----END RSA PRIVATE KEY-----

Also peculiar: GNOME somehow manages to add the key on login with seahorse. Then ssh-add -L does list the key but it is not usable:

sign_and_send_pubkey: signing failed: agent refused operation

Answer 1

Traditionally OpenSSH used the same private key format is identical to the older PEM format used by OpenSSL. (Because it uses OpenSSL for parsing the key, it will accept the newer PKCS#8 format as well.)

So the issue can be one of:

2. Your OpenSSL version refuses to load this key format. Perhaps it has accidentally enabled FIPS mode and refuses any algorithms except those part of its original FIPS validation?

   Try loading the key into the openssl command-line tool (which, yes, might also be linked to a different libcrypto, and you should check with ldd):

   openssl rsa -noout -text < id_rsa
   openssl pkey -noout -text < id_rsa
   

   Try converting it to PKCS#8 format:

   umask 077
   openssl pkey < id_rsa > id_rsa.pkcs8
   ssh-add id_rsa.pkcs8
   

3. Your OpenSSH has been built without OpenSSL support. Even though ssh -V says the support was enabled, that does not automatically mean the ssh-add binary is the same – it might come from a different partial installation.

   Use type -a ssh and type -a ssh-add to compare installation locations.

   Once you know the path, use ldd /usr/bin/ssh-add to verify that it's linked to libcrypto.so (the OpenSSL cryptographic library).

---

If nothing works at all, try converting your key to the new OpenSSH-proprietary format using... PuTTY. Install the putty package for Fedora, and use:

puttygen id_rsa -o id_rsa.newformat -O private-openssh-new
ssh-add id_rsa.newformat

---

Also peculiar: GNOME somehow manages to add the key on login with seahorse.

Older GNOME Keyring versions have an internal copy of the SSH agent code and are independent from the system OpenSSH. So they will accept keys that your OpenSSH won't. (But on the other hand, this means severe lagging in terms of feature support (such as Ed25519 keys), and the latest GNOME Keyring just uses the system ssh-agent instead.)

Answer 2

In my case problem caused incorrect end of line characters in id_rsa file, while copying my Windows text editor wanted to help me and converted EOLs to CR LF.