KOPS Kubernetes cannot login to bastion host, ssh public key permission error

Question

I am learning about Kubernetes and wanted to provision one such cluster in AWS via the KOPS tool. Followed official tutorial then for short this one too https://medium.com/andcloudio/kubernetes-kops-cluster-on-aws-f55d197d8304

I also made sure to add the ssh key before trying to connect to the bastion host as explained here too https://kops.sigs.k8s.io/bastion/#using-the-bastion

All goes fine, nodes, workes, load balancers etc get created and bastion host too.

The only problem is that I cannot ssh into the bastion host with the key. I runed the ssh with -vvv to see verbose output and log is below. I don't understand what is the problem

ssh -A admin@${bastion_elb_url} -vvv

Warning: Permanently added 'bastion-single-k8s-local-noarfe-151938406.eu-central-1.elb.amazonaws.com,3.121.65.83' (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /root/.ssh/id_rsa (0x55d6af4ea570), agent
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

I am also posting key results to help troubleshooting:

root@vagrant:/srv# ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCq9cN3EAEy0WiASY/IBkF9SPIpLv/bZt1tpLc95cb5fG++ac5VX36rA4XukJFtCAk6I4P82ysuqfZGUQNsB57yibz9rbKZ1bFfxRPyGZS22/1Omqb/8B2NlNpJx42sK4odyUj3G+KLCGCmID/AEDhbjeY7d99ZuE6g8aqrtSo0fwsmNHnpvDS8Dt0IjbLxg41Sms9tmYDLlc/tncAs9BmRvuhPbg+BDw+z7ecLneI7+TexDfhXbnZkYfjFLsfI8vWivOu8ptuGVvPkQz/MJo+MokZEzoGbVCAZP5mYSIz+LIFnnCoh5WOMsB3OZuwvelR5bBgWjQhvOaWOX8BuSU5v /root/.ssh/id_rsa

there are millions of reasons why that might not work; but what troubleshooting have you already tried on your own, to allow us to skip recommending those steps? — mdaniel, Dec 30 '20 at 19:25
I just debugged the ssh output as printed here, I do not understand why it doesn't work. When creating the cluster I specified to use the key at same path, as instructed in documentation like --ssh-public-key ~/.ssh/id_rsa.pub in kops create command. The key exists both public and private at /root/.ssh directory.Then I cannot get ssh to work after cluster and bastion successfully created. — Kristi Jorgji, Dec 30 '20 at 19:56
You misunderstand me; it's not adding more verbosity to the ssh command, it's checking that AMI to see that the user really is `admin` (which is a suspicious login for an AMI), looking at the EC2 console to ensure you and AWS are on the same page, checking the user-data to see if the key was injected that way, checking the instance console output to see if there was some other error, checking the ELB to ensure it provisioned correctly and the hosts are healthy ... I can keep going, but that's what I meant — mdaniel, Dec 31 '20 at 00:21
@mdaniel thank you! Interestingly before user was really admin (without dns gossip and bastion), now tried with ubuntu user and logged in successfully to the bastion host. Then from it had to repeat process copy keys from host -> bastion, then ssh from bastion to kubernetes master. This worked now but I expected the -A flag as doumented officially to forward somehow to the master but did not happen. Had to double ssh manually and copy keys to bastion — Kristi Jorgji, Dec 31 '20 at 19:35

score 0 · Answer 1 · answered Mar 19 '21 at 12:05

As you can see in the verbose output access was denied based on the publickey that was used when trying to authenticate:

debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

You can see above that all files that are checked by default (if the particular file wasn't specified with -i flag) were not found in your /root/.ssh/ directory.

As already discussed in comments it turned out that you used the admin user that wasn't defined on the remote host. You confirmed that with ubuntu user you managed to login successfully: "now tried with ubuntu user and logged in successfully to the bastion host" As it was clafiried sufficiently I will focus only on answering your additional question, posted in comments:

Then from it had to repeat process copy keys from host -> bastion, then ssh from bastion to kubernetes master. This worked now but I expected the -A flag as doumented officially to forward somehow to the master but did not happen. Had to double ssh manually and copy keys to bastion – Kristi Jorgji Dec 31 '20 at 19:35

The ssh login process you described is known as ssh-ing via so called a jump host. Keep in mind that it doesn't work this way out of the box and requires additional configuration. Take a look at this article, as it clearly explains everything, you need to know about setting up SSH agent forwarding which needs to be done if you want to use your local keys to ssh not only to the bastion host (which happens to be a jump host in this scenario) but also to automatically ssh from there to another remote host.

In short, you need to create ~/.ssh/config file on your local machine, if it doesn't exist, and set the host you want to allow your local ssh keys to be forwarded to and set ForwardAgent to yes:

Host example.com # it can be either domain name or IP address
  ForwardAgent yes

Additionally make sure your jump host allows SSH agent forwarding on inbound connections:

Agent forwarding may also be blocked on your server. You can check that agent forwarding is permitted by SSHing into the server and running sshd_config. The output from this command should indicate that AllowAgentForwarding is set.

Now you should be able to ssh directly to your destination remote host via a jumphost directly from your local machine with one ssh command. It's well described here:

Dynamic jumphost list

You can use the -J option to jump through a host:
user $ ssh -J host1 host2
If usernames or ports on machines differ, specify them:
user $ ssh -J user1@host1:port1 user2@host2:port2
Multiple jumps
The same syntax can be used to make jumps over multiple machines:
user $ ssh -J user1@host1:port1,user2@host2:port2 user3@host3

@KristiJorgji Does this [answer your question](https://stackoverflow.com/help/someone-answers)? — Wytrzymały Wiktor, Mar 22 '21 at 13:19

KOPS Kubernetes cannot login to bastion host, ssh public key permission error

1 Answers1

Dynamic jumphost list