SSH Agent forwarding using different usernames and different keys

Question

There is a very similar question that if answered may have been the answer to this question. Unfortunately it's a case of "no need to answer the question I asked because the problem wasn't what I thought it was."

The Setup

1. The server bastion.ec2 accepts an ssh connection from my workstation via ssh -i mykey.pem [e-mail]2
2. The server service1.ec2 accepts ssh connections only from bastion.ec2 via ssh -i sharedkey.pem [e-mail]2

The Requirements

1. Both keys are only on my workstation so I can't actually do the 2nd command without copying the key over
2. For security reasons I want to use ssh-agent forwarding rather than copy ssh keys over to bastion.ec2

The Solution

This is where you come in. How can I forward a different key for the 2nd connection?

If shareduser had mykey.pub in it's ~/.ssh/authorized_keys this would work:

ssh -i mykey.pem [e-mail]2 ssh [e-mail]2

However, I don't want every user to have to put their public key in every server.

Answer 1

Step 1

Make sure your local agent is ready

Just because you can ssh into your bastion server without specifying your key path or getting prompted for the password, doesn't mean that your ssh agent is running and holding your key. Some modern OSes (eg: OSX) handle this for you.

On your local machine

$ ssh-add -L
ssh-rsa ObahfCbvagGbLbhSbeHfvatEBG13== ~/.ssh/mykey.pem
ssh-rsa LbhNerWhfgJnnlGbbPyrireEBG13== ~/.ssh/sharedkey.pem

^fig.1

That means your agent is running and has your key.

$ ssh-add -L
The agent has no identities.

^fig.2

That means you haven't added any keys to your agent. Fix that with:

ssh-add ~/.ssh/mykey.pem ~/.ssh/sharedkey.pem

^fig.3

Step 2

Make sure your remote agent is ready

SSH into your bastion server and repeat the check from fig.1 & fig.2. However, the error you are more likely to get is this:

$ ssh-add -L
Could not open a connection to your authentication agent.

^fig.4

That most likely means that your SSH client is not forwarding your authentication agent connection.

You can force this with the -A flag (as long as the sshd config on the server allows it, which is the default).

$ ssh -A bastion.ec2

^fig.5

Step 3

Make sure you're using the right keys

If you have added keys to your agent, your agent is forwarding, and your remote agent lists your local keys. There is only two likely reasons you are not getting a connection. Either you are not using the right key or you are not using the right username.

Output the public counterpart to your private key:

$ cd
$ cd .ssh
$ ssh-keygen -y -f mykey.pem
ssh-rsa ObahfCbvagGbLbhSbeHfvatEBG13
$ ssh-keygen -y -f sharedkey.pem
ssh-rsa LbhNerWhfgJnnlGbbPyrireEBG13

^fig.6

These should be the same as what you were seeing from ssh-add -L up to the == near the end.

Now one way or another you have to get into the box you are failing to connect to and look at the content of the $HOME/.ssh/authorized_keys file for the user you are trying to connect as. You need make sure the public key you output with the command above is in that file on a line by itself. You can't trust that the sharedkey.pub that the Bro 2 cubes emailed you is right. Verify! This may require getting someone else who can SSH in as that user to get you the authorized_keys file or getting root access. If you've come this far and it still isn't working, you are beyond taking shortcuts.

Step 4

Make it easy

Hopefully the steps above have gotten you in. Now let's make this headache go away for as long as you are using this workstation.

Configure your local ssh-client

Host *
    # A lot of people put an IdentityFile line in this Host * section.
    # Don't do that unless you will use only 1 key everywhere forever.
    #IdentityFile id_rsa

Host bastion.ec2
    # You want to make sure you always forward your agent to this host.
    # But don't forward to untrusted hosts. So don't put it in Host *
    ForwardAgent yes
    # Go a head and put the IP here in case DNS ever fails you.
    # Comment it out if you want. Having it recorded is a good backup.
    HostName 172.31.0.1
    # You don't want to create a proxy loop later, so be explicit here.
    ProxyCommand none
    # SSH should try using all keys in your .ssh folder, but if you
    # know you want this key, being explicit speeds authentication.
    IdentityFile ~/.ssh/mykey.pem

# Connect effortlessly by hostname or IP address
# This assumes that your internal DNS uses the fake TLD ec2
# This assumes that 172.31.0.0 is your C-Class subnet
Host *.ec2 172.31.*
    # This command says proxy all ssh connections through bastion as if
    # you had done an ssh -A
    ProxyCommand ssh -W %h:%p bastion.ec2
    ForwardAgent yes
    # These next lines are documentation you leave as a love letter to
    # your future self when all else fails or you have to help a
    # coworker and decide to look at your own config.
    # ssh-add ~/.ssh/*.pem
    # ssh -At bastion.ecs ssh admin@172.31.18.19

^fig.7

If you take nothing else away from fig.7 it should be the proper use of ProxyCommand & ForwardAgent.

Auto populate your .bash_profile

You don't want to have to do ssh-add manually every time you log into your machine. ~/.bash_profile is a script that runs each time you login**. Put the line from fig. 3 in there and you should always have your agent ready.

** Don't confuse this with .bashrc which runs for each new [interactive] terminal. Your agent continues running even if you close all your terminal sessions. No Need to reload your keys

Alternative to using .bash_profile

I have also created a gist that add an OSX/macOS Launch Agent. You can use that method for starting your ssh-agent on boot. It is very easy to install:

curl -sSL https://gist.github.com/RichardBronosky/429a8fff2687a16959294bcee336dd2a/raw/install.sh | bash

Answer 2

This is where you come in. How can I forward a different key for the 2nd connection?

You do not forward the keys. You forward agent and it can have completely independent keys than you are using for authentication in the first jump. Check the keys in your agent using ssh-add -L.

But even better, run the connection over ProxyCommand ssh -W %h:%p [e-mail]2, which will avoid the need to forward the agent, the heap or command-line options and a need to authenticate from the immediate host.

You can simply put that in your config (see man ssh_config).