SSH through intermediate host fails only on myuser@mymac but works elsewhere

Question

I am not able to connect via ssh to one of my datacenter nodes using my user on my macbook. This is a recent problem, and it was perfectly funtional since ~ a couple of weeks ago.

Strangely, this only affects my user on my computer, but I am able to establish the connection from:

A different user in the same machine, using the same ssh keys and without any .ssh/config rule.
A different server, running either macos or ubuntu, with the same or different ssh keys.

Using my username in my computer, and same keys, I can:

Connect to the gateway host
Connect directly to the node using a VPN (unfortunately this is not a long-time solution)

I’m quite puzzled by this error. Can you help me locate the problem?

Looking at the logs, the connection with the gateway is established, but somehow fails at connecting to the node. On the client side:

⌘ ~ ❯ ssh -v -J gatekeeper@gateway ubuntu@node -i ~/.ssh/id_rsa 
OpenSSH_7.3p1, LibreSSL 2.4.1
[...]
debug1: Authentication succeeded (publickey).
Authenticated to gateway ([35.156.248.245]:22).
debug1: channel_connect_stdio_fwd node:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: client_input_global_request: rtype keepalive@openssh.com want_reply 1
channel 0: open failed: connect failed: Connection timed out
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

On the gateway side:

admin@gateway:~$ grep -e "\[7669\]" -e "\[7739\]" /var/log/auth.log
Mar 13 11:01:20 gateway sshd[7669]: Set /proc/self/oom_score_adj to 0
Mar 13 11:01:20 gateway sshd[7669]: rexec line 32: Deprecated option PermitBlacklistedKeys
Mar 13 11:01:20 gateway sshd[7669]: Connection from <laptop-out-ip> port 62113 on <gateway-ip> port 22
Mar 13 11:01:20 gateway sshd[7669]: Postponed publickey for gatekeeper from <laptop-out-ip> port 62113 ssh2 [preauth]
Mar 13 11:01:20 gateway sshd[7669]: Accepted publickey for gatekeeper from <laptop-out-ip> port 62113 ssh2: RSA 8d:7e:9c:53:11:c9:4d:b3:67:7b:ae:04:03:8f:e2:71
Mar 13 11:01:20 gateway sshd[7669]: pam_unix(sshd:session): session opened for user gatekeeper by (uid=0)
Mar 13 11:01:20 gateway sshd[7669]: User child is on pid 7739
Mar 13 11:03:27 gateway sshd[7739]: error: connect_to <node-ip> port 22: failed.
Mar 13 11:03:28 gateway sshd[7739]: Connection closed by <laptop-out-ip>
Mar 13 11:03:28 gateway sshd[7739]: Transferred: sent 2252, received 2864 bytes
Mar 13 11:03:28 gateway sshd[7739]: Closing connection to <laptop-out-ip> port 62113
Mar 13 11:03:28 gateway sshd[7669]: pam_unix(sshd:session): session closed for user gatekeeper

On the node side there is no entry in the logs.

The ssd_config at the gateway:

# ssh service configuration

AcceptEnv
AddressFamily inet
AllowAgentForwarding yes
AllowGroups
AllowTcpForwarding no
AllowUsers gatekeeper
AuthorizedKeysFile %h/.ssh/authorized_keys
ChallengeResponseAuthentication no
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
ClientAliveCountMax 3
ClientAliveInterval 15
Compression delayed
DenyGroups
DenyUsers
GSSAPIAuthentication no
GatewayPorts no
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostbasedAuthentication no
KerberosAuthentication no
ListenAddress 0.0.0.0:22
LogLevel VERBOSE
LoginGraceTime 60
MaxAuthTries 6
MaxSessions 10
MaxStartups 30
PasswordAuthentication no
PermitBlacklistedKeys no
PermitRootLogin no
PermitTunnel no
PermitUserEnvironment no
PidFile /var/run/sshd.pid
PrintLastLog yes
PrintMotd no
Protocol 2
PubkeyAuthentication yes
RSAAuthentication no
RhostsRSAAuthentication no
StrictModes yes
SyslogFacility AUTH
TCPKeepAlive yes
UseDNS no
UseLogin no
UsePAM yes
UsePrivilegeSeparation yes
X11Forwarding no

Match User gatekeeper
AllowTcpForwarding yes
AllowAgentForwarding no
X11Forwarding no

score 4 · Accepted Answer · answered Mar 21 '17 at 23:16

4

Finally I have been able to corner and identify the source of the problem. I could make the problem disappear by not sourcing the iterm2 shell integration, or just updating it to the latest version. This may be somehow related to using the fish shell.

I did not dig further into the problem, if anyone's interested please let me know.

answered Mar 21 '17 at 23:16

RogerFC

332
1
2
10

Was the shell integration problem in the jump host or the target host? – Mikko Rantalainen Sep 24 '20 at 13:19

score 2 · Answer 2 · answered Mar 13 '17 at 14:02

2

error: connect_to <node-ip> port 22: failed.

This looks like the port forwarding is disabled or limited by PermitOpen option in sshd config. Make sure it is allowed, if not, please post the sshd_config from gateway.

answered Mar 13 '17 at 14:02

Jakuje

9,145
2
40
44

I added the ssd_config for the gateway. not, though, that I am able to connect with the same command, keys, etc, from the same macbook and a different user. – RogerFC Mar 13 '17 at 14:29
AFAIK, it is the option `AllowTcpForwarding no`. – Jakuje Mar 13 '17 at 14:30
This is set to yes in the Match User gatekeeper section. – RogerFC Mar 13 '17 at 14:33
1

If it is set in the global block to the `no`, it [is NOT overwritten](http://serverfault.com/questions/833128/openssh-different-authentication-methods/833168#833168) in the following `Match` block. – Jakuje Mar 13 '17 at 14:34
From the sshd_config man page I interpret otherwise: `Match: Introduces a conditional block. If all of the criteria on the Match line are satisfied, the keywords on the following lines override those set in the global section of the config file, until either another Match line or the end of the file. If a keyword appears in multiple Match blocks that are satisfied, only the first instance of the keyword is applied.` In any case, I tried it and it still does not work. – RogerFC Mar 13 '17 at 14:51
Yes, this is probably wrong in manual page and it should be fixed since it is very confusing for users. I will fill a bug upstream, if it will solve your problems. Let me know. – Jakuje Mar 13 '17 at 14:57
Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/55284/discussion-between-rogerfc-and-jakuje). – RogerFC Mar 13 '17 at 15:03

score 1 · Answer 3 · answered Mar 13 '17 at 18:40

1

that I am able to connect with the same command, keys, etc, from the same macbook and a different user

Maybe you need to RUN this with the new user (in your home directory):

ssh-keygen -t rsa

ssh-copy-id -i .ssh/id_rsa.pub gatekeeper@gateway 

ssh-copy-id -i .ssh/id_rsa.pub ubuntu@node

answered Mar 13 '17 at 18:40

xCanox

51
5

This is not a problem with the ssh keys. They work fine used from the other account, or other laptop. – RogerFC Mar 14 '17 at 10:05
@RogerFC Did you try? – xCanox Mar 14 '17 at 12:50
I already tried using a different ssh key pair from my account, yes. – RogerFC Mar 14 '17 at 13:36

score 0 · Answer 4 · answered Jul 16 '20 at 21:10

I had something similar to this. I wasn't able to ssh through the intermediate host, but I was able to ssh into the intermediate host.

Turns out I just had an outdated ssh/config. Overwrote my ssh/config with the ssh_proxy_config that was on the intermediate host, and I was good to go

score 0 · Answer 5 · answered Sep 24 '20 at 13:44

I hit almost same problem and in my case ssh -v target emitted

(... while connecting to jumphost.example.com:)
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:REDACTED /home/USER/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 277
debug1: Authentication succeeded (publickey).
Authenticated to jumphost.example.com ([1.2.3.4]:22).
debug1: channel_connect_stdio_fwd target:1234
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
channel 0: open failed: connect failed: Connection refused
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

So similar up to debug1: pledge: network. After that I see hostkey-00 instead of keepalive and channel 0: open failed: connect failed: Connection refused.

In my case it was a typo in my .ssh/config where Port of Host target was accidentally set to 1234 instead of the correct port.

Unfortunately, the diagnostics of ProxyJump or ssh -J are seriously lacking. The feature itself is superb, though!

SSH through intermediate host fails only on myuser@mymac but works elsewhere

5 Answers5