We use SSSD to provide AD authentication, and kerberos TGT acquisition, on Centos 7.3 build 1611.
This works correctly for 99% of users most of the time, but we've hit an issue where post-password change (via Windows PC), a single user can no longer log in to Centos (but can login to Windows, and other associated AD / LDAP services - email - etc)
We've tried tracing, both SSH and SSSD, reset-ing pam_faillock entries, providing different servers (joined via realmd to the same AD domain), but we still see a message indicating the user's password is incorrect.
If we try and kinit as the failing user, that also fails with the usual message indicating password incorrectness:
kinit: Preauthentication failed while getting initial credentials
I've checked all I can really - to my untrained eye this doesn't look like a Centos / SSSD issue, but rather something central. However, have you ever tried going to AD admins with something as vague as this?! :-)
Just wondering if anyone's seen anything like it, and what if anything we can do to fix.
SSD tracing up to debug 7 - krb5_child.log:
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [main] (0x0400): krb5_child started.
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [unpack_buffer] (0x1000): total buffer size: [133]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [unpack_buffer] (0x0100): cmd [241] uid [792856944] gid [792800513] validate [true] enterprise principal [true] offline [false] UPN [<USERNAME>@<DOMAIN>]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [unpack_buffer] (0x0100): ccname: [FILE:/tmp/krb5cc_792856944] old_ccname: [not set] keytab: [/etc/krb5.keytab]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [check_use_fast] (0x0100): Not using FAST.
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [privileged_krb5_setup] (0x0080): Cannot open the PAC responder socket
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [become_user] (0x0200): Trying to become user [792856944][792800513].
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_LIFETIME] from environment.
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to [true]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [main] (0x0400): Will perform online auth
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [tgt_req_child] (0x1000): Attempting to get a TGT
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [<KRB5REALM>]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [get_and_save_tgt] (0x0020): 1296: [-1765328360][Preauthentication failed]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [map_krb5_error] (0x0020): 1365: [-1765328360][Preauthentication failed]
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [k5c_send_data] (0x0200): Received error code 1432158221
(Fri Apr 21 12:40:17 2017) [[sssd[krb5_child[2488]]]] [main] (0x0400): krb5_child completed successfully
And the SSHD logfile (with DEBUG set)
Apr 21 10:01:25 <CENTOSHOST> sshd[21720]: debug1: Forked child 21779.
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: Set /proc/self/oom_score_adj to 0
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: inetd sockets after dupping: 3, 3
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: Connection from <USERIPADDRESS> port 54908 on <LINUXHOST> port 22
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: Client protocol version 2.0; client software version PuTTY_Release_0.60
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: no match: PuTTY_Release_0.60
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: Enabling compatibility mode for protocol 2.0
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: Local version string SSH-2.0-OpenSSH_6.6.1
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: SELinux support enabled [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: permanently_set_uid: 74/74 [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_KEXINIT received [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: kex: client->server aes256-ctr hmac-sha1 none [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: kex: server->client aes256-ctr hmac-sha1 none [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32 [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32 [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_KEX_DH_GEX_REQUEST_OLD received [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
Apr 21 10:01:25 <CENTOSHOST> sshd[21779]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
Apr 21 10:01:26 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent [preauth]
Apr 21 10:01:26 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Apr 21 10:01:26 <CENTOSHOST> sshd[21779]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Apr 21 10:01:26 <CENTOSHOST> sshd[21779]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Apr 21 10:01:26 <CENTOSHOST> sshd[21779]: debug1: KEX done [preauth]
Apr 21 10:02:18 <CENTOSHOST> sshd[21779]: debug1: userauth-request for user <USERACCOUNT> service ssh-connection method none [preauth]
Apr 21 10:02:18 <CENTOSHOST> sshd[21779]: debug1: attempt 0 failures 0 [preauth]
Apr 21 10:02:18 <CENTOSHOST> sshd[21779]: debug1: PAM: initializing for "<USERACCOUNT>"
Apr 21 10:02:18 <CENTOSHOST> sshd[21779]: debug1: PAM: setting PAM_RHOST to "<USERPC>"
Apr 21 10:02:18 <CENTOSHOST> sshd[21779]: debug1: PAM: setting PAM_TTY to "ssh"
Apr 21 10:02:18 <CENTOSHOST> sshd[21779]: debug1: userauth_send_banner: sent [preauth]
Apr 21 10:02:24 <CENTOSHOST> sshd[21779]: debug1: userauth-request for user <USERACCOUNT> service ssh-connection method password [preauth]
Apr 21 10:02:24 <CENTOSHOST> sshd[21779]: debug1: attempt 1 failures 0 [preauth]
Apr 21 10:02:24 <CENTOSHOST> sshd[21779]: pam_succeed_if(sshd:auth): requirement "user in <LOCALSUPERACCOUNT>" not met by user "<USERACCOUNT>"
Apr 21 10:02:27 <CENTOSHOST> sshd[21779]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<USERPC> user=<USERACCOUNT>
Apr 21 10:02:27 <CENTOSHOST> sshd[21779]: pam_sss(sshd:auth): received for user <USERACCOUNT>: 17 (Failure setting user credentials)
Apr 21 10:02:29 <CENTOSHOST> sshd[21779]: debug1: PAM: password authentication failed for <USERACCOUNT>: Authentication failure
Apr 21 10:02:29 <CENTOSHOST> sshd[21779]: Failed password for <USERACCOUNT> from <USERIPADDRESS> port 54908 ssh2
Apr 21 10:05:42 <CENTOSHOST> sshd[21779]: Connection closed by <USERIPADDRESS> [preauth]
Apr 21 10:05:42 <CENTOSHOST> sshd[21779]: debug1: do_cleanup [preauth]
Apr 21 10:05:42 <CENTOSHOST> sshd[21779]: debug1: monitor_read_log: child log fd closed
Apr 21 10:05:42 <CENTOSHOST> sshd[21779]: debug1: do_cleanup
Apr 21 10:05:42 <CENTOSHOST> sshd[21779]: debug1: PAM: cleanup
Apr 21 10:05:42 <CENTOSHOST> sshd[21779]: debug1: Killing privsep child 21780
Thanks. Any advice gratefully received.
