I hope this is the correct forum to ask.
We run a cluster (Centos 7) using FreeIPA for account management. On Sunday the IPA server suddenly restarted and since then, users are no longer able to login via ssh and Kerberos credentials can no longer be requested successfully:
$ KRB5_TRACE=/dev/stdout kinit
[29387] 1658843092.500360: Getting initial credentials for user@ABC.UNI-XX.DE
[29387] 1658843092.500362: Sending unauthenticated request
[29387] 1658843092.500363: Sending request (195 bytes) to ABC.UNI-XX.DE
[29387] 1658843092.500364: Resolving hostname ceg-ipa01.abc.uni-xx.de
[29387] 1658843092.500365: Initiating TCP connection to stream XXX.XXX.XXX.XXX:88
[29387] 1658843092.500366: Terminating TCP connection to stream XXX.XXX.XXX.XXX:88
[29387] 1658843092.500367: Sending initial UDP request to dgram XXX.XXX.XXX.XXX:88
kinit: Cannot contact any KDC for realm 'ABC.UNI-XX.DE' while getting initial credentials
Here, ABC.UNI-XX.DE
is the realm and ceg-ipa01
hosts the IPA server. Pinging ceg-ipa01
works fine. I already restarted ceg-ipa01
and the krb5kdc
and kadmin
services, without any success. I can, however, access the cluster using the local accounts like root.
This is the krb5.conf
file on the IPA server
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = ABC.UNI-XX.DE
dns_lookup_realm = false
dns_lookup_kdc = true
rdns = false
ticket_lifetime = 24h
forwardable = true
udp_preference_limit = 0
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
ABC.UNI-XX.DE = {
kdc = ceg-ipa01.abc.uni-xx.de:88
master_kdc = ceg-ipa01.abc.uni-xx.de:88
admin_server = ceg-ipa01.abc.uni-xx.de:749
default_domain = abc.uni-xx.de
pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}
[domain_realm]
.abc.uni-xx.de = ABC.UNI-XX.DE
abc.uni-xx.de = ABC.UNI-XX.DE
ceg-ipa01.abc.uni-xx.de = ABC.UNI-XX.DE
[dbmodules]
ABC.UNI-XX.DE = {
db_library = ipadb.so
}
[plugins]
certauth = {
module = ipakdb:kdb/ipadb.so
enable_only = ipakdb
}
and this is the same file on the clients
#File modified by ipa-client-install
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/
[libdefaults]
default_realm = ABC.UNI-XX.DE
dns_lookup_realm = false
dns_lookup_kdc = false
rdns = false
dns_canonicalize_hostname = false
ticket_lifetime = 30d
forwardable = true
udp_preference_limit = 0
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
ABC.UNI-XX.DE = {
kdc = ceg-ipa01.abc.uni-xx.de:88
master_kdc = ceg-ipa01.abc.uni-xx.de:88
admin_server = ceg-ipa01.abc.uni-xx.de:749
kpasswd_server = ceg-ipa01.abc.uni-xx.de:464
default_domain = abc.uni-xx.de
pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}
[domain_realm]
.abc.uni-xx.de = ABC.UNI-XX.DE
abc.uni-xx.de = ABC.UNI-XX.DE
ceg-octane.abc.uni-xx.de = ABC.UNI-XX.DE
As far as I see, there is no obvious syntax error (like missing capitalization).
If that's of any help, we also run LDAP with the following configuration on ceg-ipa01
$ cat /etc/openldap/ldap.conf
SASL_NOCANON on
URI ldaps://ceg-ipa01.abc.uni-xx.de
BASE dc=abc,dc=uni-xx,dc=de
TLS_CACERT /etc/ipa/ca.crt
SASL_MECH GSSAPI
and on the clients
$ cat /etc/openldap/ldap.conf
TLS_CACERTDIR /etc/openldap/cacerts
# Turning this off breaks GSSAPI used with krb5 when rdns = false
SASL_NOCANON on
#URI ldaps://ceg-ipa01.abc.uni-xx.de # modified by IPA
URI ldap://ldap
#BASE dc=abc,dc=uni-xx,dc=de # modified by IPA
BASE dc=abc,dc=uni-xx,dc=de
TLS_CACERT /etc/ipa/ca.crt
I also see this error message in /var/messages
:
failed to bind to LDAP server ldap://ceg-ipa01.abc.uni-xx.de: Can't contact LDAP server: Transport endpoint is not connected
and running ldapsearch
on the client yields
# ldapsearch -x -b "dc=abc,dc=uni-xx,dc=de" -d-1 -H ldap://ceg-ipa01.abc.uni-xx.de -v
ldap_url_parse_ext(ldap://ceg-ipa01.abc.uni-xx.de)
ldap_initialize( ldap://ceg-ipa01.abc.uni-xx.de:389/??base )
ldap_create
ldap_url_parse_ext(ldap://ceg-ipa01.abc.uni-xx.de:389/??base)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP ceg-ipa01.abc.uni-xx.de:389
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying XXX.XXX.XXX.XXX:389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
attempting to connect:
connect errno: 113
ldap_close_socket: 3
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
Bottom line, the clients cannot properly connect to the server and hence, the authentication fails. But as I am new to FreeIPA, Kerberos and LDAP, I do not know where to look further or what to do now. So I really appreciate any help!