7

Frustratingly, SFTP users suddenly stopped being able to connect to my Amazon Linux server.

The /var/log/secure shows the following error:

sshd[7291]: fatal: safely_chroot: stat("/chroot/uhleeka"): Permission denied [postauth]

/var/log/secure:

==> /var/log/secure <==
Nov 21 23:49:23 amzl-lamp sshd[7291]: Accepted password for uhleeka from 172.31.0.254 port 47170 ssh2
Nov 21 23:49:24 amzl-lamp sshd[7291]: pam_unix(sshd:session): session opened for user uhleeka by (uid=0)
Nov 21 23:49:24 amzl-lamp sshd[7291]: fatal: safely_chroot: stat("/chroot/uhleeka"): Permission denied [postauth]
Nov 21 23:49:24 amzl-lamp sshd[7291]: pam_unix(sshd:session): session closed for user uhleeka

Debug output from SSHD:

$ /usr/sbin/sshd -ddd -p 33333
...
debug1: SELinux support disabled
debug1: PAM: establishing credentials
debug3: PAM: opening session
debug1: monitor_reinit: /dev/log doesn't exist in /chroot/%u chroot - will try to log via monitor using [postauth] suffix
User child is on pid 6655
debug1: PAM: establishing credentials [postauth]
debug3: safely_chroot: checking '/' [postauth]
debug3: safely_chroot: checking '/chroot/' [postauth]
debug3: safely_chroot: checking '/chroot/uhleeka' [postauth]
safely_chroot: stat("/chroot/uhleeka"): Permission denied [postauth]
debug1: do_cleanup [postauth]
debug3: PAM: sshpam_thread_cleanup entering [postauth]
debug3: mm_request_send entering: type 124 [postauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 124
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
debug3: PAM: sshpam_thread_cleanup entering

SELinux is disabled:

$ sestatus
SELinux status:                 disabled

$ ls -lZ /chroot/uhleeka/
drwxr-x--- root root      ?                                .
drwx--x--- root sftp-only ?                                ..
drwx--x--- root sftp-only ?                                etc
drwxr-xr-x root sftp-only ?                                home

There have been no configuration changes or permissions changes, but yum update was run yesterday:

$ rpm -qa --last
system-release-2016.09-0.8.noarch             Mon 21 Nov 2016 04:34:40 PM UTC
cloud-init-0.7.6-2.14.amzn1.noarch            Mon 21 Nov 2016 04:34:40 PM UTC
python26-botocore-1.4.74-1.60.amzn1.noarch    Mon 21 Nov 2016 04:34:39 PM UTC
openssh-server-6.6.1p1-31.62.amzn1.x86_64     Mon 21 Nov 2016 04:34:39 PM UTC
openssh-clients-6.6.1p1-31.62.amzn1.x86_64    Mon 21 Nov 2016 04:34:39 PM UTC
aws-cli-1.11.17-1.43.amzn1.noarch             Mon 21 Nov 2016 04:34:39 PM UTC
python27-botocore-1.4.74-1.60.amzn1.noarch    Mon 21 Nov 2016 04:34:38 PM UTC
bind-utils-9.8.2-0.47.rc1.51.amzn1.x86_64     Mon 21 Nov 2016 04:34:38 PM UTC
bind-libs-9.8.2-0.47.rc1.51.amzn1.x86_64      Mon 21 Nov 2016 04:34:38 PM UTC
openssh-6.6.1p1-31.62.amzn1.x86_64            Mon 21 Nov 2016 04:34:37 PM UTC
...

Regarding the openssh update: https://alas.aws.amazon.com/ALAS-2016-770.html

It was discovered that the OpenSSH sshd daemon fetched PAM environment settings before running the login program. In configurations with UseLogin=yes and the pam_env PAM module configured to read user environment settings, a local user could use this flaw to execute arbitrary code as root.

/etc/ssh/sshd_config has:

UsePAM yes
#UseLogin no
#PermitUserEnvironment no

The latest updates seem like the most likely culprit. Is there a configuration issue that would cause only chroot users to suddenly stop be denied access with the latest openssh?

uhleeka
  • 173
  • 1
  • 5
  • Is selinux enabled and enforcing? If so see /var/log/audit/audit.log. – Mark Wagner Nov 22 '16 at 01:06
  • What are the permissions on the `/chroot/uhleeka`? Add the output of `ls -lZ /chroot/uhleeka`. – Jakuje Nov 22 '16 at 08:57
  • @MarkWagner SELinux is disabled. – uhleeka Nov 22 '16 at 14:16
  • @Jakuje Output added, but SELinux is disabled, so no context is available. – uhleeka Nov 22 '16 at 14:17
  • That is weird, what amazon does with these updates. I have no idea why the `stat` should start failing from nowhere. – Jakuje Nov 22 '16 at 15:19
  • Having the same problem, updated on the 18th. – w00t Nov 23 '16 at 12:56
  • 2
    @uhleeka This worked for me: rpm -e --nodeps openssh openssh-server openssh-clients; yum install openssh-server-0:6.6.1p1-25.61.amzn1.x86_64 openssh-clients-0:6.6.1p1-25.61.amzn1.x86_64 – w00t Nov 23 '16 at 13:14
  • @w00t Thanks. I rolled back as well. I don't have UseLogin=yes on my sshd_config, so I'm comfortable not running the latest patch. Hopefully it gets fixed in the next version, or guidance is provided on how the chroot configuration has changed. – uhleeka Nov 23 '16 at 16:27

2 Answers2

7

Edit: This should be fixed in openssh-6.6.1p1-32.el7 per https://bugzilla.redhat.com/show_bug.cgi?id=1398569

It appears after the OpenSSH-6.6.1p1-31 update, only the user's primary group is checked for authentication during the SFTP connection attempt. With root and the user's primary group owning the home directory and at least 710 permissions, connection attempts should succeed.

Repro steps: 

$ groups sftpuser  
sftpuser : sftpgroup sftpuser  
$ ls -ld /home/sftpuser/  
drwx--x--- 2 root sftpuser 4096 Nov 22 18:31 sftpuser/  
$ sftp sftpuser@localhost  
sftpuser@localhost's password:  
Write failed: Broken pipe  
Couldn't read packet: Connection reset by peer  
$ chgrp sftpgroup sftpuser/  
$ ls -ld /home/sftpuser/  
drwx--x--- 2 root sftpgroup 4096 Nov 22 18:31 sftpuser/  
$ sftp sftpuser@localhost  
sftpuser@localhost's password:  
Connected to localhost.  
sftp> exit

Failed connection with secondary group owning home directory (from /var/log/secure):

sshd[31640]: Accepted password for sftpuser from 127.0.0.1 port 34380 ssh2
sshd[31640]: pam_unix(sshd:session): session opened for user sftpuser by (uid=0)
sshd[31640]: fatal: Unable to chdir to chroot path "/home/sftpuser": Permission denied [postauth]
sshd[31640]: pam_unix(sshd:session): session closed for user sftpuser
Successful connection with primary group owning home directory (from /var/log/secure):
sshd[31647]: Accepted password for sftpuser from 127.0.0.1 port 34382 ssh2
sshd[31647]: pam_unix(sshd:session): session opened for user sftpuser by (uid=0)
sshd[31647]: session opened for local user sftpuser from [127.0.0.1] [postauth]
sshd[31647]: session closed for local user sftpuser from [127.0.0.1] [postauth]
sshd[31647]: Received disconnect from 127.0.0.1: 11: disconnected by user [postauth]
sshd[31647]: pam_unix(sshd:session): session closed for user sftpuser
Will
  • 86
  • 3
  • 2
    Thanks. I can confirm that this works for me. How did you find this to be the solution? – uhleeka Nov 24 '16 at 06:24
  • Glad to hear that helped. Found it through lots of trial and error. Confirmed to also occur on RHEL 7.3. – Will Nov 24 '16 at 17:02
  • I have myuser:bla on /var/sftp/bla and bla's primary group is bla, perms 750 and it still failed. – w00t Nov 24 '16 at 21:22
  • Root needs to be the user owner of bla's chroot home directory. – Will Nov 25 '16 at 15:33
  • @w00t I have the same issue here. – Joris Mans Nov 29 '16 at 08:51
  • @Will yes of course. I typed the wrong stuff, it's actually /var/sftp/bla/bla where /var/sftp/bla is root:root chmod 755, and /var/sftp/bla/bla is myuser:bla chmod 750. sshd_config has ChrootDirectory /var/sftp/%u. – w00t Nov 29 '16 at 16:35
  • @Joris Mans I ended up downgrading openssh as my setup is in production with a lot of clients and couldn't spend time fixing it. – w00t Nov 29 '16 at 16:36
  • @w00t With bla being the user and /var/sftp/%u (/var/sftp/bla) being the chroot home directory, the owners for /var/sftp/bla would need to be root:bla'sPrimaryGroup and permissions of at least 710. – Will Nov 29 '16 at 19:32
  • Interesting, so this got changed from needing to be root:root (I believe this was previously required) to root:User'sPrimaryGroup – w00t Nov 29 '16 at 19:42
0

I posted a similar question after reading the answer from @Will. I tried that solution but could not get it right for my scenario.

Permissions on chrooted user not working after update on Amazon Linux

But it led me to the solution - change the primary GID for the user accounts I chrooted to the GID of the group I used to create permissions in the chrooted directory. It worked.

Community
  • 1
Complectia
  • 1
  • 2