NFS/krb5 authentication server lookup fails due to wrong principal name

Question

When mounting an NFSv4 with Kerberos, authentication fails and krb5kdc.log shows the wrong principal name for the NFS server.

LOOKING_UP_SERVER: ... host/nfsclient.internal.domain.tld@IPA.DOMAIN.TLD for nfs/containershost.internal.domain.tld@IPA.DOMAIN.TLD
... Server not found in Kerberos database

Principal nfs/containershost.internal.domain.tld should be nfs/nfs.internal.domain.tld.

Am I correct in assuming that a reverse DNS query is being performed which is returning containershost.internal.domain.tld? If so, at what step in the process (and from which machine) does this lookup happen?

$ dig -x 192.111.111.111
111.111.111.192.in-addr.arpa. 6009 IN PTR containershost.internal.domain.tld.

Is there a way to prevent this reverse DNS query from happening? The host at IP 192.111.111.111 (containershost.internal.domain.tld) is running multiple containers and so adding PTR records isn't possible.

Docker Container Host

Hostname: containershost.internal.domain.tld
IP: 192.111.111.111

FreeIPA Server (docker)

Image: freeipa/freeipa-server:centos-8-4.8.4
Container Name: freeipa
Container Host: containershost.internal.domain.tld
Hostname: freeipa.internal.domain.tld
IP: 172.222.222.222
Domain: ipa.domain.tld
Realm: IPA.DOMAIN.TLD
Keytab:

$ klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 host/freeipa.internal.domain.tld@IPA.DOMAIN.TLD

/etc/krb5.conf:

[libdefaults]
    default_realm = IPA.DOMAIN.TLD
    dns_lookup_realm = false
    dns_lookup_kdc = true
    rdns = false
    ticket_lifetime = 24h
    forwardable = true
    udp_preference_limit = 0

[realms]
    IPA.DOMAIN.TLD = {
        kdc = freeipa.internal.domain.tld:88
        master_kdc = freeipa.internal.domain.tld:88
        admin_server = freeipa.internal.domain.tld:749
        default_domain = ipa.domain.tld
        pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
        pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
    }

[domain_realm]
    .ipa.domain.tld = IPA.DOMAIN.TLD
    ipa.domain.tld = IPA.DOMAIN.TLD
    freeipa.internal.domain.tld = IPA.DOMAIN.TLD
    .internal.domain.tld = IPA.DOMAIN.TLD
    internal.domain.tld = IPA.DOMAIN.TLD

NFS Server (docker)

Image: ubuntu:latest
Container Name: nfs
Container Host: containershost.internal.domain.tld
Hostname: nfs.internal.domain.tld
IP: 172.333.333.333
Running Services:

- /usr/sbin/rpc.mountd --port 32767 --no-nfs-version 2 --no-nfs-version 3 -F --debug all
- /usr/sbin/rpc.idmapd -S -vvv -f
- /usr/sbin/rpc.nfsd --debug --port 2049 --no-nfs-version 2 --no-nfs-version 3 -L 10 -G 10
- /usr/sbin/rpc.svcgssd -f -vvv -rrr -iii -p nfs/nfs.internal.domain.tld

Keytab:

$ klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
  74 host/nfs.internal.domain.tld@IPA.DOMAIN.TLD
  66 nfs/nfs.internal.domain.tld@IPA.DOMAIN.TLD

/etc/krb5.conf:

[libdefaults]
    default_realm = IPA.DOMAIN.TLD
    dns_lookup_realm = false
    dns_lookup_kdc = false
    rdns = false

[realms]
    IPA.DOMAIN.TLD = {
        kdc = freeipa
        admin_server = freeipa
        default_domain = domain.tld
    }

[domain_realm]
    .domain.tld = IPA.DOMAIN.TLD
    domain.tld = IPA.DOMAIN.TLD

NFS Client

OS: Ubuntu 20.04
Hostname: nfsclient.internal.domain.tld
IP: 192.444.444.444
Keytab:

$ klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
  5 host/nfsclient.internal.domain.tld@IPA.DOMAIN.TLD

Mount Command:

$ sudo mount -vvv -t nfs4 -o sec=krb5p 192.111.111.111:/ /mountpoint
mount.nfs4: trying text-based options 'sec=krb5,vers=4.2,addr=192.111.111.111,clientaddr=192.444.444.444'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting 192.111.111.111:/

Result

/var/log/krb5kdc.log:

freeipa krb5kdc[288](info): TGS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), 
DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23)}) 192.444.444.444: LOOKING_UP_SERVER: authtime 0, 
etypes {rep=(0)} host/nfsclient.internal.domain.tld@IPA.DOMAIN.TLD for nfs/containershost.internal.domain.tld@IPA.DOMAIN.TLD, 
Server not found in Kerberos database

Answer 1

The nfs server resolves it's own name to containershost.internal.domain.tld. Thus, it looks for that name in keytab file. You probably should explicitly set the container name to nfs.internal.domain.tld (with -name option or host directive in docker-compose) or add yet another entry into keytab file.