0

I'm running a Kerberos / LDAP authentication server for many years. Kerberos data is stored inside LDAP. Now, I have a second site and want to mirror the server to the new site. This basically works, but there is a strange side effect. Each server has a KDC and a LDAP running. KDC talks to LDAP using local ldapi:///.

If I use the original KDC krb1.example.com I can authenticate to the master LDAP and the replica. If I use the replicated KDC krb2.example.com, I can still authenticate to the master LDAP, but trying the replica I get

SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
        additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (KDC has no support for encryption type)

Which is strange, since krb2 is literally a clone on the LXC container krb1 i.e. the configurations are identical safe for the changes required for replication. But still more puzzeling is a look into the ticket caches after trying to query either LDAP server. With a TGT from krb1

root@krb2:/# klist -e
Ticket cache: FILE:/tmp/krb5_cc_0.tkt
Default principal: user@EXAMPLE.COM

Valid starting       Expires              Service principal
04.02.2022 01:05:29  04.02.2022 11:05:29  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 05.02.2022 01:05:26, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
04.02.2022 01:05:42  04.02.2022 11:05:29  ldap/krb1.example.com@EXAMPLE.COM
        renew until 05.02.2022 01:05:26, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
04.02.2022 01:05:53  04.02.2022 11:05:29  ldap/krb2.example.com@EXAMPLE.COM
        renew until 05.02.2022 01:05:26, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

and from krb2:

root@krb2:/# klist -e
Ticket cache: FILE:/tmp/krb5_cc_0.tkt
Default principal: user@EXAMPLE.COM

Valid starting       Expires              Service principal
04.02.2022 00:53:45  04.02.2022 10:53:45  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 05.02.2022 00:53:41, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
04.02.2022 00:53:47  04.02.2022 10:53:45  ldap/krb1.example.com@EXAMPLE.COM
        renew until 05.02.2022 00:53:41, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

which has no entry for ldap/krb2.example.com due to the error. But apparently there is no difference in the encryption types required. So, why does it complain?

Currently both LDAP servers refer to krb1. But due to the LDAP replication all keys should be identical i.e. a key from krb1 should be the same as from krb2, shouldn't it? And if keys from krb1 work for both LDAP servers, why does the key from the replica server only work for the master LDAP?

supported_enctypes for the realm in both /etc/krb5kdc/kdc.conf are supported_enctypes = aes256-cts:normal arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3. There are no enctype directives in either /etc/krb5.conf. There are no ssf configuration used in either LDAP. The krbPrincipalName entries for krb2 exist in both LDAP.

Even if I obtain the TGT from krb2 and then switch the krb5.conf to krb1 for the service ticket, I can authenticate to LDAP on krb2.

OpenLDAP is in version 2.4.47, Kerberos 1.17 on both machines (current Debian stable).

Update: It turned out that replication is incomplete i.e., krbPrincipalKey is not (reliably) replicated. I checked the slapd debugging output:

Feb  9 19:14:52 krb1 slapd[19560]: conn=1300 op=3 BIND authcid="sync/krb2.example.com@EXAMPLE.COM" authzid="dn:uid=sync/krb2.example.com,cn=example.com,cn=gssapi,cn=auth@EXAMPLE.COM"
Feb  9 19:14:52 krb1 slapd[19560]: conn=1300 op=3 BIND dn="cn=admin,dc=example,dc=com" mech=GSSAPI sasl_ssf=256 ssf=256

So, synprov apparently binds as cn=admin,dc=example,dc=com, which is intended.However, if I do

root@krb1:~# ldapsearch -b 'cn=KERBEROS,dc=example,dc=com' -D 'cn=admin,dc=example,dc=com' -Wx 'krbPrincipalName=ldap/krb2.example.com@EXAMPLE.COM'

then I see krbPrincipalKey. So why doesn't it get replicated?

I restart slapd on krb2 with options -c rid=1,csn=0, which I understood should sync the entire database. But since some entries have a key and some don't, I do not trust that this is true.

Lars Hanke
  • 281
  • 2
  • 15
  • What exactly do you mean by "Currently both LDAP servers refer to krb1"? Are you talking about KDCs in krb5.conf, or about keytabs, or something else? – user1686 Feb 05 '22 at 14:08
  • Can you show the actual authentication process that is failing? If the containers are exact clones, does the LDAP service on kdc2 _know_ that it's now called `ldap/kdc2` rather than `ldap/kdc1`, i.e. does it at least have a new keytab for its new name, or is it still using the original keytab? – user1686 Feb 05 '22 at 14:10
  • Yes, hostname, IP address, DNS entry, keytab, TLS certificates have been adapted to match the new instance. – Lars Hanke Feb 07 '22 at 09:04
  • So what I did is: ldapsearch -b "cn=admin,dc=example,dc=com" -H ldap://kdc?.uac.microsult.de. Setting /etc/krb5.conf to kdc2, I can do kinit, I can query kdc1, but kdc2 fails. If I then change kdc to krb1 in /etc/krb5.conf, the query to kdc2 succeeds as well using the very same ticket. – Lars Hanke Feb 07 '22 at 09:10
  • Can you manually request a ticket from the 'kdc2' KDC using `KRB5_TRACE=/dev/stderr kvno ldap/kdc2.uac.microsult.de`, and if that fails, what does krb5kdc report to its log file? Can you verify that both KDCs have the same information in their LDAP replicas (e.g. by dumping them both via slapcat, or via ldapsearch with rootdn/rootpw authentication)? Can you run `kadmin.local` in both KDCs to check what they know about the ldap/kdc2 principal? – user1686 Feb 07 '22 at 09:34
  • Ah, and you really shouldn't have that "supported_enctypes" setting... it probably doesn't _hurt_ (yet), but the list is already quite stale, and having it explicitly configured doesn't serve any good purpose. – user1686 Feb 07 '22 at 09:36
  • The ```kadmin.local``` hint showed that the replica server actually has no keys for some principals. I didn't see this since my admin principal for LDAP neither sees keys due to ACL i.e., the entries looked the same on both servers. However, the replication principal should see keys. So it seems I have to troubleshoot the ```slapd``` ACL and Authzid. I did not yet find any suitable debug value to see, which users use which authzid to which DN. But this at least is a pure LDAP configuration issue. – Lars Hanke Feb 09 '22 at 18:06
  • `stats` log level would be a good start. – user1686 Feb 09 '22 at 18:08

1 Answers1

0

The problem was solved by slapcat the master database and slapadd from scratch on the consumer.

Thanks to user1686 for pointing to the things to check to make me break circular thoughts.

The cause of the issue was that some Kerberos principals in the LDAP had no krbPrincipalKey attribute. ldap/krb2.example.com being one of those. This became obvious using get_principal in kadmin.local on each server. The confusion arose due to using inappropriate bind DN and the ACL associated with those. Furthermore, I trusted that I used options for slapd to re-fetch the entire database on start-up, which did not happen.

Some pitfalls that kept me trapped: Since with TLS enabled -Y EXTERNAL does not work, I use special Kerberos principals for database maintenance. My principal for cn=config has no access to credentials stored in the main database, which I was not aware of anymore. So comparing entries of the two LDAP yielded identical results. While the actual replication principal has access to credentials, I probably used the other bind DN initially i.e., before I had Kerberos set up for replication. Therefore, the entries created during this time period were replicated without credentials, and syncprov does not fix this later.

Lars Hanke
  • 281
  • 2
  • 15