I have a server that authenticates SSH through RSA key pairs. Yesterday my rsync backup worked successfully I know this because of log files:
/var/log/backups_log$ ls -l
total 815536
-rw-r--r-- 1 root root 139244471 Jul 26 20:02 2019-07-26_backup_log.log
-rw-r--r-- 1 root root 139076680 Jul 27 20:01 2019-07-27_backup_log.log
-rw-r--r-- 1 root root 139197173 Jul 28 20:01 2019-07-28_backup_log.log
-rw-r--r-- 1 root root 139249372 Jul 29 20:02 2019-07-29_backup_log.log
-rw-r--r-- 1 root root 141445775 Jul 30 20:11 2019-07-30_backup_log.log
-rw-r--r-- 1 root root 136870570 Jul 31 20:06 2019-07-31_backup_log.log
Today when I'm trying to SSH into my backup server I'm getting an error that looks like this:
ssh -p50683 10.0.1.41
Ubuntu 16.04.6 LTS
Permission denied (publickey).
On the backup server I get the following error from the auth.log file:
~/.ssh$ tail -f -n0 /var/log/auth.log
Aug 1 12:43:03 Hljoo sshd[666]: Connection from 10.0.1.68 port 53690 on 10.0.1.41 port 50683
Aug 1 12:43:03 Hljoo sshd[666]: User root not allowed because account is locked
Aug 1 12:43:03 Hljoo sshd[666]: input_userauth_request: invalid user root [preauth]
Aug 1 12:43:03 Hljoo sshd[666]: Connection closed by 10.0.1.68 port 53690 [preauth]
You'd think it would be because the account is locked, but I also get the error when attempting to login to an account that is authorized, but isn't root:
~/.ssh$ tail -f -n0 /var/log/auth.log
Aug 1 12:44:00 Hljoo sshd[671]: Connection from 10.0.1.68 port 53704 on 10.0.1.41 port 50683
Aug 1 12:44:00 Hljoo sshd[671]: Invalid user username from 10.0.1.68
Aug 1 12:44:00 Hljoo sshd[671]: input_userauth_request: invalid user username [preauth]
Aug 1 12:44:00 Hljoo sshd[671]: Connection closed by 10.0.1.68 port 53704 [preauth]
Now comes the fun part. I figured maybe it had something to do with my RSA pairs, so I regenerated them successfully and transferred the keys over to my backup server. However, after doing so I get the same error. Incase you need it, here's my configuration file:
# Package generated configuration file
# See the sshd_config(5) manpage for details
# What ports, IPs and protocols we listen for
Port 50683
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes
# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024
# Logging
SyslogFacility AUTH
LogLevel VERBOSE
# Authentication:
LoginGraceTime 120
PermitRootLogin prohibit-password
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes
# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no
MaxAuthTries 3
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no
#MaxStartups 10:30:60
Banner /etc/issue.net
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM no
Has anyone experienced anything similar to this? Also if so, how did you fix it?
Requested information:
ls -al /root/.ssh
total 24
drwx------ 2 root root 4096 May 16 04:32 .
drwx------ 19 root root 4096 Aug 1 09:04 ..
-rw-r--r-- 1 root root 403 Apr 29 02:00 authorized_keys
-rw------- 1 root root 1679 Aug 1 12:25 id_rsa
-rw-r--r-- 1 root root 396 Aug 1 12:25 id_rsa.pub
-rw-r--r-- 1 root root 888 Jul 29 08:02 known_hosts
passwd -S root
root L 05/25/2018 0 99999 7 -1
pam_tally2 --user=root
Login Failures Latest failure From
root 0
ssh -v -p50683 10.0.1.41
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.0.1.41 [10.0.1.41] port 50683.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.0.1.41:50683 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:haAvEjO8pjy5QKLsGPAKqQe3n2AUKb3L5gRt0obkImI
debug1: checking without port identifier
debug1: Host '10.0.1.41' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: found matching key w/out port
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
Ubuntu 16.04.6 LTS
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
