PAM Authentication on AD joined Linux server intermittently works after joining new server

Question

We are joining RHEL 7.3 servers to a Windows Active Directory server (Windows 2012 R2 DC) using Winbind. We are able to successfully join a Linux server to the domain 100% of the time. The server shows up in the Windows active directory, and we are able to query the directory.

The issue is that most of the time the domain credentials will not work immediately, we are given an "Access is Denied" error from SSH.. We have had to wait anywhere from 10 minutes, to 4-5 hours before domain credentials would work. In some cases, we would restart winbind which would immediately fix the problem, but this did not always resolve the authentication issue. We've also attempted full reboots.

Domain values obfuscated for security.

[root@acmeprodweb01 ~]# realm list
domain.com
  type: kerberos
  realm-name: DOMAIN.COM
  domain-name: domain.com
  configured: kerberos-member
  server-software: active-directory
  client-software: winbind
  required-package: oddjob-mkhomedir
  required-package: oddjob
  required-package: samba-winbind-clients
  required-package: samba-winbind
  required-package: samba-common-tools
  login-formats: DOMAIN+%U
  login-policy: allow-any-login

Domain User:

[root@acmeprodweb01 ~]# id DOMAIN+user
uid=201604(DOMAIN+user) gid=200513(DOMAIN+domain users) groups=200513(DOMAIN+domain users),201604(DOMAIN+user),214118(DOMAIN+linux-sysadmin-tools),214138(DOMAIN+usr_localadmin)

For what it's worth, I can make tickets with the domain controller, after logging in as a local account.:

Default principal: [e-mail]

Valid starting       Expires              Service principal
02/17/2017 21:35:25  02/18/2017 07:35:25  krbtgt/[e-mail]
    renew until 02/23/2017 21:35:25

This is the error we receive when trying to log into a server that is rejecting domain credentials:

Feb 17 20:47:55 acmeprodweb01 sshd[2218]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.46.129.254  user=DOMAIN+user
Feb 17 20:47:55 acmeprodweb01 sshd[2218]: pam_winbind(sshd:auth): getting password (0x00000390)
Feb 17 20:47:55 acmeprodweb01 sshd[2218]: pam_winbind(sshd:auth): pam_get_item returned a password
Feb 17 20:47:55 acmeprodweb01 sshd[2218]: pam_winbind(sshd:auth): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_SYSTEM_ERR (4), NTSTATUS: NT_STATUS_ACCESS_DENIED, Error message was: Access denied
Feb 17 20:47:55 acmeprodweb01 sshd[2218]: pam_winbind(sshd:auth): internal module error (retval = PAM_SYSTEM_ERR(4), user = 'DOMAIN+user')

Here is our configuration file:

[global]
workgroup = DOMAIN
security = ads

    passdb backend = tdbsam

    printing = cups
    printcap name = cups
    load printers = yes
    cups options = raw
kerberos method = system keytab
template homedir = /home/%D/%U
template shell = /bin/bash
realm = DOMAIN.COM
#idmap backend = tdb
#idmap gid = 10000-2000000
#idmap uid = 10000-2000000
winbind use default domain = no
winbind refresh tickets = yes
winbind offline logon = yes
winbind enum groups = no
winbind enum users = no
winbind separator = +
idmap config * : backend = rid
idmap config * : range = 200000-299999
idmap config * : rangesize = 10000
idmap config INTMGMT : backend = rid
idmap config INTMGMT : range = 100000-199999
idmap config INTMGMT : rangesize = 10000
password server = domain.com
allow trusted domains = yes
log level = 10
debug pid = true
max log size = 0

[homes]
    comment = Home Directories
    valid users = %S, %D%w%S
    browseable = No
    read only = No
    inherit acls = Yes

[printers]
    comment = All Printers
    path = /var/tmp
    printable = Yes
    create mask = 0600
    browseable = No

[print$]
    comment = Printer Drivers
    path = /var/lib/samba/drivers
    write list = root
    create mask = 0664
    directory mask = 0775

And our pam.d system-auth config:

auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        sufficient    pam_winbind.so cached_login use_first_pass
auth required pam_faillock.so preauth audit silent deny=3 unlock_time=900
auth [success=1 default=bad] pam_unix.so
auth [default=die] pam_faillock.so authfail audit deny=3 unlock_time=900
auth sufficient pam_faillock.so authsucc audit deny=3 unlock_time=900
auth        required      pam_deny.so

account     required      pam_unix.so broken_shadow
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     [default=bad success=ok user_unknown=ignore] pam_winbind.so cached_login
account     required      pam_permit.so

# implements CIS 5.3.1, GISOD-PR Retry Rule
password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=

# implement CIS 5.3.3, GISOD-PR History Rule, CIS 5.3.4
password    sufficient    pam_unix.so shadow nullok try_first_pass use_authtok remember=24 sha512
password    sufficient    pam_winbind.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     optional      pam_oddjob_mkhomedir.so umask=0077
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_winbind.so cached_login warn_pwd_expire=7

Hopefully this is enough information, if not, please feel free to ask.

Answer 1

It turns out the issue was on the domain controller itself, not on our Winbind configuration.

Our secondary domain controller did not have netlogon or sysvol shared. So when a request hit the primary domain controller, the authentication was successful, but if the request hit the secondary, there was no response.

The issue was resolved once we enabled netlogon and sysvol domain shares on the secondary DC.