34

I have no clue on how this happens. The distro is Scientific Linux 6.1 and everything is set up to perform authentication via public key. Yet, when sshd is running as a daemon (service sshd start), it doesn't accept public keys. (To obtain this piece of log, I've changed the sshd script to add the -ddd option)

debug1: trying public key file /root/.ssh/authorized_keys
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys2
debug1: restore_uid: 0/0
Failed publickey for root from xxx.xxx.xxx.xxx port xxxxx ssh2
debug3: mm_answer_keyallowed: key 0x7f266e1a8840 is not allowed
debug3: mm_request_send entering: type 22
debug3: mm_request_receive entering
debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa
debug3: Wrote 64 bytes for a total of 1853
debug1: userauth-request for user root service ssh-connection method publickey
debug1: attempt 2 failures 1

If sshd is run in debug mode /usr/sbin/sshd -ddd, authentication works like a charm:

debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 1
Found matching RSA key: d7:3a:08:39:f7:28:dc:ea:f3:71:7c:23:92:02:02:d8
debug1: restore_uid: 0/0
debug3: mm_answer_keyallowed: key 0x7f85527ef230 is allowed
debug3: mm_request_send entering: type 22
debug3: mm_request_receive entering
debug3: Wrote 320 bytes for a total of 2109
debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa
Postponed publickey for root from xxx.xxx.xxx.xxx port xxxxx ssh2
debug1: userauth-request for user root service ssh-connection method publickey
debug1: attempt 2 failures 0

Any ideas?? Has anyone seen anything like this?

Notes:

File permissions have been double checked:

# ll -d .ssh
drwx------. 2 root root 4096 Oct 14 10:05 .ssh
# ll .ssh
total 16
-rw-------. 1 root root  786 Oct 14 09:35 authorized_keys
-rw-------. 1 root root 1675 Oct 13 08:24 id_rsa
-rw-r--r--. 1 root root  393 Oct 13 08:24 id_rsa.pub
-rw-r--r--. 1 root root  448 Oct 13 12:51 known_hosts

I was asked if sshd can access root's files in "daemon mode". The closest answer I get to this question is:

# netstat -ntap | grep 22
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      19847/sshd 
# ps -ef | grep 19847
root     19847     1  0 09:58 ?        00:00:00 /usr/sbin/sshd

If sshd is running as root, I don't know how it's not possible to access its own files. Could SELinux be the cause?

masegaloeh
  • 17,978
  • 9
  • 56
  • 104
user666412
  • 443
  • 1
  • 4
  • 7
  • 1
    Does the sshd init script do anything interesting? (Should be /etc/init.d/sshd?) that you're not doing on the command line? Instead of 'service sshd start' try 'sh -x /etc/init.d/ssh start'. – P.T. Oct 14 '11 at 17:18

4 Answers4

44

Yes, SELinux is likely the cause. The .ssh dir is probably mislabeled. Look at /var/log/audit/audit.log. It should be labeled ssh_home_t. Check with ls -laZ. Run restorecon -r -vv /root/.ssh if need be.

Mark Wagner
  • 17,764
  • 2
  • 30
  • 47
  • 1
    Yep, SELinux was the cause: type=AVC msg=audit(1318597097.413:5447): avc: denied { read } for pid=19849 comm="sshd" name="authorized_keys" dev=dm-0 ino=262398 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file It works after running "restorecon -r -vv /root/.ssh". Thanks a lot. – user666412 Oct 14 '11 at 19:44
  • 1
    thanks THANKS THANKS for the selinux command line fix I've been trying to find for ages why it was I could ssh as root to my redhat enterprise 6.2 server using ssh key authentication, but I couldn't ssh in as a non-root user without having to enter a password. "ssh -v" didn't indicate anything unusual at all. I'd checked and rechecked the file protections on /home/example/.ssh It wasn't until I ran "/usr/sbin/sshd -d" and for some reason that worked normally that I realised something else was happening, and tried a different google search and found this. So, the symptoms were I could ssh as ro – Paul M Jun 29 '12 at 13:05
  • Thank you! This was driving me crazy! For posterity, can you explain better what `restorecon -r -vv /root/.ssh` is doing to fix this? – Stefan Lasiewski Sep 24 '13 at 19:05
  • Thanks, neither ssh -vvv nor LogLevel VERBOSE in sshd.conf points to SELinux. – count0 Jun 02 '14 at 20:52
  • 1
    I had to do this on the whole filesystem, i.e. `restorecon -r /`, YMMV. – Irfy Oct 15 '14 at 11:18
  • NO - not working. I also tried this but still failing with same status. http://unix.stackexchange.com/q/208152/11085 – YumYumYum Jun 10 '15 at 10:20
  • 1
    I tried this - but still not working . in the audit log I have `type=AVC msg=audit(1434642809.455:94717): avc: denied { search } for pid=27032 comm="sshd" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=dir` - not sure what it means – Yehosef Jun 18 '15 at 07:56
  • 1
    The answer was in the `name="/"` - I had to run the `restorecon -r /` as @Irfy suggested. – Yehosef Jun 18 '15 at 09:39
  • `restorecon ~/.ssh` didn't fix it for me. I think the "label" is stuck and isn't going to change (I have a strange user home dir setup). Luckily, I am able to disable SELinux so that fixed it for me! (SELINUX=disabled in /etc/selinux/config) – harperville Apr 13 '16 at 14:38
  • @Irfy @Yehosef I found the same Error Message in my `audit.log` on each `sshd` Restart. I also run the Command `restorecon -r /` as suggested. This situation came up because I enabled SELinux afterwards the System Installation was completed. So it was not correctly setup in the System. Thank you everyone! – Bodo Hugo Barwich Jul 04 '18 at 10:10
  • Been working with Linux for quite some time now, lots with CentOS servers, and always when there is some inexplicable problem with those, the culprit was SELinux 9/10 times. – kasimir Apr 02 '20 at 13:28
3

I had the same issue. In my case, restorecon and chcon did not work.

I did not want to disable selinux. After lots of research, I finally figured it was because my home directory was mounted from elsewhere (NFS). I found this bug report which clued me in.

I ran:

> getsebool use_nfs_home_dirs
use_nfs_home_dirs --> off

to confirm use_nfs_home_dirs was off and then:

sudo setsebool -P use_nfs_home_dirs 1

to turn it on.

Now I can ssh to my machine using my key and without entering a password. Toggling the use_home_nfs_dirs boolean was what it took for me.

Andrew Schulman
  • 8,561
  • 21
  • 31
  • 47
Gerard
  • 31
  • 1
2

To add to Mark Wagner's answer, if your're using a custom home directory path (i.e. not /home), you need make sure you've set the SELinux security context. To do so, if you have user home directories in, for example, /myhome, run:

semanage fcontext -a -e /home /myhome
restorecon -vR /myhome
DavidArndt
  • 121
  • 2
  • If you're on CentOS, you'll need to run this to get `semanage`: `sudo yum install policycoreutils-python` – sm4rk0 Nov 07 '18 at 09:45
0

It looks like you use different keys when testing the connections, 0x7f266e1a8840 vs 0x7f85527ef230. Try connecting using 'ssh -v example.com' to sshd running as a daemon and in debug mode and look for the keys used by ssh around the string "Offering RSA public key".

Johan Nilsson
  • 348
  • 1
  • 6
  • Yes, there were id_rsa and id_dsa. DSA key is gone and I'll redo the test. – user666412 Oct 14 '11 at 19:16
  • The value mentioned in `debug3: mm_answer_keyallowed: key 0xFFFFFFFFFF` will change every time the sshd receives a new connection. To confim this, find a server where SSH does work, crank up the sshd LOGLEVEL to debug3, restart sshd, run `tail -f /var/log/secure |grep mm_answer_keyallowed` and then log in a few times, waiting a few seconds (or minutes) between each connection. You'll see that the value changes each time. And actually it looks like a counter to me. – Stefan Lasiewski Sep 24 '13 at 18:47