NFS4 + Kerberos does not work since 5.10 kernel

Question

Since I updated to Debian Bullseye, nfs clients stopped working:

# mount -vvt nfs4 -o sec=krb5 nfs11:/srv /mnt
mount.nfs4: timeout set for Wed Sep 15 20:25:49 2021
mount.nfs4: trying text-based options 'sec=krb5,vers=4.2,addr=x.y.11.63,clientaddr=x.y.11.42'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting nfs11:/srv

When I install 5.9 kernel (linux-image-5.9.0-0.bpo.5-cloud-amd64) on the same system - it works.

I also tried:

Debian Testing Kernel (kernel 5.14) - does not work
Ubuntu 21.10 Impish (kernel 5.13) - does not work
Ubuntu 20.04 Focal (kernel 5.4) - works

Provided, that all the systems has an identical NFS/Kerberos setup, my conclusion: Something has changed in the kernel that does not allow to mount NFS/Kerberos shares.

My KDC - Samba4 AD
My Kerberos and NFS setup is pretty mach standard, as in any how-to
HOSTNAME$@REALM nfs/fqdn@REALM host/... principles are there in the client and server key-tab

I put RPCGSSDOPTS="-vvv" in /etc/default/nfs-common for debugging. In the following logs:

nfs11 - my test nfs server (Debian 11, kernel 5.10)
tst2 - my test nfs client (Debian 11)

Here is the syslog when the client is attempting to mount nfs share:

nfs client booted with 5.9 kernel (mounts successfully)

rpc.gssd[446]: #012handle_gssd_upcall: 'mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2' (nfs/clnt0)
rpc.gssd[446]: krb5_use_machine_creds: uid 0 tgtname (null)
rpc.gssd[446]: Full hostname for 'nfs11.my.domain' is 'nfs11.my.domain'
rpc.gssd[446]: Full hostname for 'tst2.my.domain' is 'tst2.my.domain'
rpc.gssd[446]: Success getting keytab entry for 'tst2$@MY.DOMAIN'
rpc.gssd[446]: gssd_get_single_krb5_cred: principal 'tst2$@MY.DOMAIN' ccache:'FILE:/tmp/krb5ccmachine_MY.DOMAIN'
rpc.gssd[446]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631755378
rpc.gssd[446]: creating tcp client for server nfs11.my.domain
rpc.gssd[446]: DEBUG: port already set to 2049
rpc.gssd[446]: creating context with server nfs@nfs11.my.domain
rpc.gssd[446]: doing downcall: lifetime_rec=36000 acceptor=nfs@nfs11.my.domain
rpc.gssd[446]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2' (nfs/clnt0)
rpc.gssd[446]: krb5_use_machine_creds: uid 0 tgtname (null)
rpc.gssd[446]: Full hostname for 'nfs11.my.domain' is 'nfs11.my.domain'
rpc.gssd[446]: Full hostname for 'tst2.my.domain' is 'tst2.my.domain'
rpc.gssd[446]: Success getting keytab entry for 'tst2$@MY.DOMAIN'
rpc.gssd[446]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631755378
rpc.gssd[446]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631755378
rpc.gssd[446]: creating tcp client for server nfs11.my.domain
rpc.gssd[446]: DEBUG: port already set to 2049
rpc.gssd[446]: creating context with server nfs@nfs11.my.domain
rpc.gssd[446]: doing downcall: lifetime_rec=36000 acceptor=nfs@nfs11.my.domain
nfsidmap[524]: key: 0x3b88d120 type: uid value: root@my.domain timeout 600
nfsidmap[524]: nfs4_name_to_uid: calling nsswitch->name_to_uid
nfsidmap[524]: nss_getpwnam: name 'root@my.domain' domain 'my.domain': resulting localname 'root'
nfsidmap[524]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
nfsidmap[524]: nfs4_name_to_uid: final return value is 0
nfsidmap[525]: key: 0x317cb571 type: gid value: root@my.domain timeout 600
nfsidmap[525]: nfs4_name_to_gid: calling nsswitch->name_to_gid
nfsidmap[525]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
nfsidmap[525]: nfs4_name_to_gid: final return value is 0

nfs client booted with 5.10 kernel (does not mount)

rpc.gssd[450]: #012handle_gssd_upcall: 'mech=krb5 uid=0 service=* enctypes=18,17,16,3,1,2' (nfs/clnt3)
rpc.gssd[450]: krb5_use_machine_creds: uid 0 tgtname (null)
rpc.gssd[450]: Full hostname for 'nfs11.my.domain' is 'nfs11.my.domain'
rpc.gssd[450]: Full hostname for 'tst2.my.domain' is 'tst2.my.domain'
rpc.gssd[450]: Success getting keytab entry for 'tst2$@MY.DOMAIN'
rpc.gssd[450]: gssd_get_single_krb5_cred: principal 'tst2$@MY.DOMAIN' ccache:'FILE:/tmp/krb5ccmachine_MY.DOMAIN'
rpc.gssd[450]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631656676
rpc.gssd[450]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631629984
rpc.gssd[450]: creating tcp client for server nfs11.my.domain
rpc.gssd[450]: DEBUG: port already set to 2049
rpc.gssd[450]: creating context with server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_MY.DOMAIN for server nfs11.my.domain
rpc.gssd[450]: creating tcp client for server nfs11.my.domain
rpc.gssd[450]: DEBUG: port already set to 2049
rpc.gssd[450]: creating context with server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_MY.DOMAIN for server nfs11.my.domain
rpc.gssd[450]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs11.my.domain
rpc.gssd[450]: Full hostname for 'nfs11.my.domain' is 'nfs11.my.domain'
rpc.gssd[450]: Full hostname for 'tst2.my.domain' is 'tst2.my.domain'
rpc.gssd[450]: Success getting keytab entry for 'tst2$@MY.DOMAIN'
rpc.gssd[450]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631656676
rpc.gssd[450]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631656676
rpc.gssd[450]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_MY.DOMAIN' are good until 1631629984
rpc.gssd[450]: creating tcp client for server nfs11.my.domain
rpc.gssd[450]: DEBUG: port already set to 2049
rpc.gssd[450]: creating context with server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_MY.DOMAIN for server nfs11.my.domain
rpc.gssd[450]: creating tcp client for server nfs11.my.domain
rpc.gssd[450]: DEBUG: port already set to 2049
rpc.gssd[450]: creating context with server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs11.my.domain
rpc.gssd[450]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_MY.DOMAIN for server nfs11.my.domain
rpc.gssd[450]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs11.my.domain
rpc.gssd[450]: doing error downcall

I googled a lot and didn't find anything related... Currently as a workaround I run backported kernel from previous release in all nfs client systems. But I think it is dangerous, and something tells me it may break at any time.

Has anyone experienced such a problem? Maybe I should adjust something to match changes in kernel? Maybe I should fill the kernel bug?

UPDATE. Added KDC logs.

KDC while mounting from client with 5.9 kernel - successfully

[2021/09/21 21:55:12.061264,  3] ../../source4/smbd/service_stream.c:67(stream_terminate_connection)
  stream_terminate_connection: Terminating connection - 'dcesrv: NT_STATUS_CONNECTION_DISCONNECTED'
[2021/09/21 21:55:44.743415,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: AS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:38701 for krbtgt/MY.DOMAIN@MY.DOMAIN
[2021/09/21 21:55:44.747105,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client sent patypes: 150, 149
[2021/09/21 21:55:44.747154,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for PKINIT pa-data -- tst2$@MY.DOMAIN
[2021/09/21 21:55:44.747178,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for ENC-TS pa-data -- tst2$@MY.DOMAIN
[2021/09/21 21:55:44.747209,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: No preauth found, returning PREAUTH-REQUIRED -- tst2$@MY.DOMAIN
[2021/09/21 21:55:44.751030,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: AS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:50506 for krbtgt/MY.DOMAIN@MY.DOMAIN
[2021/09/21 21:55:44.753959,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client sent patypes: encrypted-timestamp, 150, 149
[2021/09/21 21:55:44.754060,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for PKINIT pa-data -- tst2$@MY.DOMAIN
[2021/09/21 21:55:44.754114,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for ENC-TS pa-data -- tst2$@MY.DOMAIN
[2021/09/21 21:55:44.754187,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: ENC-TS Pre-authentication succeeded -- tst2$@MY.DOMAIN using arcfour-hmac-md5
[2021/09/21 21:55:44.754275,  3] ../../auth/auth_log.c:635(log_authentication_event_human_readable)
  Auth: [Kerberos KDC,ENC-TS Pre-authentication] user [(null)]\[tst2$@MY.DOMAIN] at [Tue, 21 Sep 2021 21:55:44.754261 +06] with [arcfour-hmac-md5] status [NT_STATUS_OK] workstation [(null)] remote host [ipv4:x.y.11.42:50506] became [MYDOM]\[tst2$] [S-1-5-21-3408476796-3867293677-901807371-6619]. local host [NULL] 
  {"timestamp": "2021-09-21T21:55:44.754359+0600", "type": "Authentication", "Authentication": {"version": {"major": 1, "minor": 2}, "eventId": 4624, "logonId": "dd24014b273cc7a8", "logonType": 3, "status": "NT_STATUS_OK", "localAddress": null, "remoteAddress": "ipv4:x.y.11.42:50506", "serviceDescription": "Kerberos KDC", "authDescription": "ENC-TS Pre-authentication", "clientDomain": null, "clientAccount": "tst2$@MY.DOMAIN", "workstation": null, "becameAccount": "tst2$", "becameDomain": "MYDOM", "becameSid": "S-1-5-21-3408476796-3867293677-901807371-6619", "mappedAccount": "tst2$", "mappedDomain": "MYDOM", "netlogonComputer": null, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", "netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "passwordType": "arcfour-hmac-md5", "duration": 3366}}
[2021/09/21 21:55:44.761108,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: AS-REQ authtime: 2021-09-21T21:55:44 starttime: unset endtime: 2021-09-22T07:55:44 renew till: 2021-09-22T21:55:44
[2021/09/21 21:55:44.761282,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client supported enctypes: arcfour-hmac-md5, aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96, using arcfour-hmac-md5/arcfour-hmac-md5
[2021/09/21 21:55:44.761368,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Requested flags: renewable-ok, forwardable
[2021/09/21 21:55:44.767382,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: TGS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:39570 for nfs/nfs11.my.domain@MY.DOMAIN [canonicalize, renewable, forwardable]
[2021/09/21 21:55:44.773999,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: TGS-REQ authtime: 2021-09-21T21:55:44 starttime: 2021-09-21T21:55:44 endtime: 2021-09-22T07:55:44 renew till: 2021-09-22T21:55:44
[2021/09/21 21:55:44.774695,  3] ../../source4/smbd/service_stream.c:67(stream_terminate_connection)
  stream_terminate_connection: Terminating connection - 'kdc_tcp_call_loop: tstream_read_pdu_blob_recv() - NT_STATUS_CONNECTION_DISCONNECTED'

KDC while mounting from client with 5.10 kernel - failed to mount

[2021/09/22 00:31:39.893723,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: AS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:46094 for krbtgt/MY.DOMAIN@MY.DOMAIN
[2021/09/22 00:31:39.899112,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client sent patypes: 150, 149
[2021/09/22 00:31:39.899162,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for PKINIT pa-data -- tst2$@MY.DOMAIN
[2021/09/22 00:31:39.899186,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for ENC-TS pa-data -- tst2$@MY.DOMAIN
[2021/09/22 00:31:39.899221,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: No preauth found, returning PREAUTH-REQUIRED -- tst2$@MY.DOMAIN
[2021/09/22 00:31:39.901942,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: AS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:39303 for krbtgt/MY.DOMAIN@MY.DOMAIN
[2021/09/22 00:31:39.905030,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client sent patypes: encrypted-timestamp, 150, 149
[2021/09/22 00:31:39.905080,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for PKINIT pa-data -- tst2$@MY.DOMAIN
[2021/09/22 00:31:39.905105,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Looking for ENC-TS pa-data -- tst2$@MY.DOMAIN
[2021/09/22 00:31:39.905171,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: ENC-TS Pre-authentication succeeded -- tst2$@MY.DOMAIN using arcfour-hmac-md5
[2021/09/22 00:31:39.905270,  3] ../../auth/auth_log.c:635(log_authentication_event_human_readable)
  Auth: [Kerberos KDC,ENC-TS Pre-authentication] user [(null)]\[tst2$@MY.DOMAIN] at [Wed, 22 Sep 2021 00:31:39.905248 +06] with [arcfour-hmac-md5] status [NT_STATUS_OK] workstation [(null)] remote host [ipv4:x.y.11.42:39303] became [MYDOM]\[tst2$] [S-1-5-21-3408476796-3867293677-901807371-6621]. local host [NULL] 
  {"timestamp": "2021-09-22T00:31:39.905331+0600", "type": "Authentication", "Authentication": {"version": {"major": 1, "minor": 2}, "eventId": 4624, "logonId": "8511280d720bd92c", "logonType": 3, "status": "NT_STATUS_OK", "localAddress": null, "remoteAddress": "ipv4:x.y.11.42:39303", "serviceDescription": "Kerberos KDC", "authDescription": "ENC-TS Pre-authentication", "clientDomain": null, "clientAccount": "tst2$@MY.DOMAIN", "workstation": null, "becameAccount": "tst2$", "becameDomain": "MYDOM", "becameSid": "S-1-5-21-3408476796-3867293677-901807371-6621", "mappedAccount": "tst2$", "mappedDomain": "MYDOM", "netlogonComputer": null, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", "netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "passwordType": "arcfour-hmac-md5", "duration": 3429}}
[2021/09/22 00:31:39.912509,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: AS-REQ authtime: 2021-09-22T00:31:39 starttime: unset endtime: 2021-09-22T10:31:39 renew till: 2021-09-23T00:31:39
[2021/09/22 00:31:39.912597,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client supported enctypes: arcfour-hmac-md5, aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96, using arcfour-hmac-md5/arcfour-hmac-md5
[2021/09/22 00:31:39.912663,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Requested flags: renewable-ok, forwardable
[2021/09/22 00:31:39.918313,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: TGS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:59850 for nfs/nfs11.my.domain@MY.DOMAIN [canonicalize, renewable, forwardable]
[2021/09/22 00:31:39.924869,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: TGS-REQ authtime: 2021-09-22T00:31:39 starttime: 2021-09-22T00:31:39 endtime: 2021-09-22T10:31:39 renew till: 2021-09-23T00:31:39
[2021/09/22 00:31:39.925340,  3] ../../source4/smbd/service_stream.c:67(stream_terminate_connection)
  stream_terminate_connection: Terminating connection - 'kdc_tcp_call_loop: tstream_read_pdu_blob_recv() - NT_STATUS_CONNECTION_DISCONNECTED'
[2021/09/22 00:31:39.928319,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: TGS-REQ tst2$@MY.DOMAIN from ipv4:x.y.11.42:59852 for nfs/nfs11.my.domain@MY.DOMAIN [renewable, forwardable]
[2021/09/22 00:31:39.930936,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Server (nfs/nfs11.my.domain@MY.DOMAIN) has no support for etypes
[2021/09/22 00:31:39.930998,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Failed building TGS-REP to ipv4:x.y.11.42:59852
[2021/09/22 00:31:39.931336,  3] ../../source4/smbd/service_stream.c:67(stream_terminate_connection)
  stream_terminate_connection: Terminating connection - 'kdc_tcp_call_loop: tstream_read_pdu_blob_recv() - NT_STATUS_CONNECTION_DISCONNECTED'

I see Server (nfs/nfs11.my.domain@MY.DOMAIN) has no support for etypes error. Google finds an old issues related to old enctypes, nothing useful. All the packages are up to date.

I have made some progress, thanks to the comments. I installed fresh Samba DC, joined both client (5.10 kernel) and server to new KDC - and it worked! New KDC allows NFS clients with any kernel to mount shares. It seems that the problem is in my production Samba DC. I looked into ldap DBs and it seems they are similar, except very few additions on fresh dc like 3 new objects and some fields. Currently I don't know what I should tweak in the production DC to make it behave like fresh one. Reinstall would be the last resort, because it involves A LOT of time.

Production DC was created long ago, and was migrated several times using standard samba replication or backups. Production and fresh DC info:

oEMInformation: Provisioned by SAMBA 4.1.6-Ubuntu
oEMInformation: Provisioned by SAMBA 4.13.5-Debian

Currently DCs are running under identical Debian operating systems.

UPDATE 2. Solved!

See the solution is below.

Can you try `sysctl sunrpc.rpc_debug=0xFFFF` to get more logs? Also, is the `cts` crypto module available? — user1686, Sep 21 '21 at 18:16
@MichaelHampton, Thank you for your suggestion! it allowed me to make some progress. I updated the post. I didn't include server logs as they do not contain errors or warnings. — Alek_A, Sep 21 '21 at 22:02
@user1686, Thanks for the Idea, I did it as well as nfsd debug, but unfortunately logs don't express that something is wrong. Can you explain what is the `cts` module and how do I check if it is available? — Alek_A, Sep 21 '21 at 22:03
Not sure whether related to your found solution, but the "Server (nfs/nfs11.my.domain@MY.DOMAIN) has no support for etypes" message sounds extremely like your NFS server's AD account is missing the "Supports AES128/AES256" parameters being enabled, so the KDC thinks it's RC4-only (which is soon to become obsolete). — user1686, Sep 22 '21 at 04:27
(I'm not sure what you meant by "joined client to new DC" though – did you set up a completely new domain? You should have a new DC to the same domain, then demoted & nuked the old DC, keeping the exact same directory contents that way but still getting a clean DC with AES support.) — user1686, Sep 22 '21 at 04:30
@user1686 As for `Server .. has no support for etypes` I think it is misleading message, because I doubt that fresh NFS server with the default settings under most recent and updated OS has problems with etypes. Could you tell me how to check If server supports AES? But It's probably related to some internal DC error caused by old LDAP parameters. "joined client to new DC" - I meant literally joined those hosts to brand new dc with `msktutil`, dc is totally new without keeping LDAP content. No need to demote and nuke old one, they may coexist. — Alek_A, Sep 22 '21 at 09:24
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/129872/discussion-between-user1686-and-alek-a). — user1686, Sep 22 '21 at 13:46

score 2 · Answer 1 · answered Dec 30 '21 at 14:56

Linux removed support for RC4-HMAC-MD5 from Kerberos in 5.10. Your client uses that encryption type as can bee seen in the server's log output:

[2021/09/21 21:55:44.761282,  3] ../../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Client supported enctypes: arcfour-hmac-md5, aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96, using arcfour-hmac-md5/arcfour-hmac-md5

If AES types were available Samba should have chosen aes256-cts-hmac-sha1-96.

It's not in any of your logs but I would guess the failing TGS-REQ asks for des3-cbc-sha1, aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96. This can be verified by starting rpc.gssd with the parameters -vvvrr. In that case the client's AD account does not have the required encryption types enabled. This will happen if the client joined the domain at a time when Samba did not support AES. You can enable the encryption types by resetting the client's AD account's password or rejoining the domain. You will also need to make sure the encryption types are added to the client's keytab. This can be verified running klist -ke on the client.

In case you use specific service principals make sure to explicitly add the encryption types to the client's account (on the ADC run net ads enctypes set <ACCOUNTNAME> 24). Otherwise only the ARCFOUR types will be exported.

I'm glad that your answer helps people, judging by the upvoting ;) I surely did what you suggested and much more. Unfortunately only solution in my specific case was that I described. Probably it's because the domain was provisioned long time ago, or because of the fact that it once was renamed. So fixing LDAP attributes of the DC helped. Anyway thanks for your time! — Alek_A, Jan 16 '22 at 17:28

Alek_A · Accepted Answer · 2021-09-22T08:50:14.277

The solution in my case was the following: I tried making LDAP DB on my production DC look like LDAP DB on the fresh DC (which is working). So I changed some fields. Rebooted everything. And it worked!

What I changed exactly.

I added/changed the following fields in the object dn: DC=my,DC=domain using ldbedit -H /var/lib/samba/private/sam.ldb:

msDS-Behavior-Version: 4
msDS-NcType: 0
serverState: 1

The production DC was renamed in the past, but I found leftovers (old names) in the following objects:

dn: CN=<old-name>,CN=*,CN=ypServ30,CN=RpcServices,CN=System,DC=my,DC=domain

I fixed this by renaming them with ldbrename, for example:

ldbrename -H /var/lib/samba/private/sam.ldb 'CN=<old-name>,CN=bootparams,CN=ypServ30,CN=RpcServices,CN=System,DC=my,DC=domain' 'CN=<actual-name>,CN=bootparams,CN=ypServ30,CN=RpcServices,CN=System,DC=my,DC=domain'

Maybe not all these changes were necessary, but it works now. Thank you for your comments!

NFS4 + Kerberos does not work since 5.10 kernel

UPDATE. Added KDC logs.

UPDATE 2. Solved!

2 Answers2