SSH by private IP is fine
I'm able to connect to a server through SSH by its private IP address:
C:\Users\m3>ssh -vvvvA uconn@192.168.1.11
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug3: Failed to open file:C:/Users/m3/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolve_canonicalize: hostname 192.168.1.11 is address
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.1.11 [192.168.1.11] port 22.
debug1: Connection established.
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_rsa type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ecdsa-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519 error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ed25519 type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.10 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.1.11:22 as 'uconn'
debug3: hostkeys_foreach: reading file "C:\\Users\\m3/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\m3/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 192.168.1.11
debug3: Failed to open file:C:/Users/m3/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:eyPiBvKLgJOk1xJc0k6cx9UnwIXbUUaXu9pPHTKt5Rg
debug3: hostkeys_foreach: reading file "C:\\Users\\m3/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\m3/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 192.168.1.11
debug3: Failed to open file:C:/Users/m3/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug1: Host '192.168.1.11' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\m3/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug3: unable to connect to pipe \\\\.\\pipe\\openssh-ssh-agent, error: 2
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug2: key: C:\\Users\\m3/.ssh/id_rsa (0000000000000000)
debug2: key: C:\\Users\\m3/.ssh/id_dsa (0000000000000000)
debug2: key: C:\\Users\\m3/.ssh/id_ecdsa (0000000000000000)
debug2: key: C:\\Users\\m3/.ssh/id_ed25519 (0000000000000000)
debug2: key: C:\\Users\\m3/.ssh/id_xmss (0000000000000000)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\m3/.ssh/id_rsa
debug3: no such identity: C:\\Users\\m3/.ssh/id_rsa: No such file or directory
debug1: Trying private key: C:\\Users\\m3/.ssh/id_dsa
debug3: no such identity: C:\\Users\\m3/.ssh/id_dsa: No such file or directory
debug1: Trying private key: C:\\Users\\m3/.ssh/id_ecdsa
debug3: no such identity: C:\\Users\\m3/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: C:\\Users\\m3/.ssh/id_ed25519
debug3: no such identity: C:\\Users\\m3/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: C:\\Users\\m3/.ssh/id_xmss
debug3: no such identity: C:\\Users\\m3/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug3: failed to open file:C:/dev/tty error:3
debug1: read_passphrase: can't open /dev/tty: No such file or directory
uconn@192.168.1.11's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (password).
Authenticated to 192.168.1.11 ([192.168.1.11]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug1: console supports the ansi parsing
debug3: Successfully set console output code page from:437 to 65001
debug3: Successfully set console input code page from:437 to 65001
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug3: unable to connect to pipe \\\\.\\pipe\\openssh-ssh-agent, error: 2
debug1: ssh_get_authentication_socket: No such file or directory
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-206-generic i686)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
0 packages can be updated.
0 of these updates are security updates.
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Tue Mar 23 14:22:05 2021 from 192.168.1.52
SSH by public IP is bad
However, when using its public IP address, I run into an error:
ssh_exchange_identification: Connection closed by remote host
C:\Users\m3>ssh -vvvvA uconn@11.111.11.111
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug3: Failed to open file:C:/Users/m3/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolve_canonicalize: hostname 11.111.11.111 is address
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 11.111.11.111 [11.111.11.111] port 22.
debug1: Connection established.
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_rsa type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_rsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_dsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ecdsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ecdsa-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519 error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ed25519 type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_ed25519-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/m3/.ssh/id_xmss-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\m3/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
ssh_exchange_identification: Connection closed by remote host
How to debug
What could be the cause? How can I debug the issue?
Router port forwarding
The server has the private IP address. But there is a router, with public IP address, which forwards the SSH 22 port to the private IP address.
sshd log
Suggested here, I used this command on server to log sshd output:
$ tail -f -n 500 /var/log/auth.log | grep 'sshd'
When I run ssh uconn@192.168.1.11 on client I get the following log:
Mar 23 17:26:10 server-homeshine sshd[1355]: Accepted password for uconn from 192.168.1.52 port 53107 ssh2
Mar 23 17:26:10 server-homeshine sshd[1355]: pam_unix(sshd:session): session opened for user uconn by (uid=0)
But when I run ssh uconn@11.111.11.111 on client, no log is displayed whatsoever. I think it is implied that the router is not forwarding the 22 port when a public IP address is used. Not sure why.
SSHD config
sshd config on server is:
uconn@server-homeshine:/etc/ssh$ cat sshd_config
# Package generated configuration file
# See the sshd_config(5) manpage for details
# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
ListenAddress ::
ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes
# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024
# Logging
SyslogFacility AUTH
LogLevel INFO
# Authentication:
LoginGraceTime 120
PermitRootLogin prohibit-password
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes
# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no
#MaxStartups 10:30:60
#Banner /etc/issue.net
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
IP tables
Here is the IP tables on server:
$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
$ sudo ip6tables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
Routing table
Server routing table:
$ sudo route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 enp9s0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 enp9s0
Wireshark/Tshark
Installing tshark on server and examining the network packets, shows that when running ssh uconn@192.168.1.11 (private IP) on a client, the SSH packets are received by server.
But when running ssh uconn@11.111.11.111 (public IP) on a client, no SSH packet is received by server whatsoever.
The conclusion is that the ADSL router is not forwarding SSH packets to the server.
ZyXEL inquiry
Contacted ZyXEL on this issue and received this reply:
When doing port forwarding you need to make sure that the internal server has a gateway address pointing back to the gateway to be able to respond to outside request.
Also check that port 22 is not used by the gateway itself.
Server default gateway
My server default gateway is 192.168.1.1 which is what it should be:
$ ip r
default via 192.168.1.1 dev enp9s0 onlink
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.11
