2

I'm trying to mount a Kerberos authenticated NFS share on a CentOS 6.4 machine. I have tried exporting the protected share from both another CentOS 6.4 machine and from our NetApp with the same results.

The CentOS machines and the NetApp are all joined to our Active Directory (2012) domain. I can SSH in to the CentOS machines using AD credentials. Tools such as kinit, msktuil, etc work. I have used mskutil to generate the keytab file for the client. The machine account in AD has principal records for the machine, root, nfs, etc. Clocks are all synced to the domain controllers.

I see in the rpc.gssd output (below) that it is not finding a key, but then it goes on to find the root key. The next line seems to show that it did NOT find the key it said it found in the previous line?

Can the hive mind help clue me in here? I feel that I'm this close to having things working...

The relevant bit of the /etc/krb5.keytab file on the client looks like this:

5 12/02/13 16:14:14 srred1kt01$@WORK.LOCAL
5 12/02/13 16:14:14 root/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:14 root/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 root/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 NFS/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 NFS/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 NFS/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 NFS/srred1kt01@WORK.LOCAL
5 12/02/13 16:14:15 NFS/srred1kt01@WORK.LOCAL
5 12/02/13 16:14:15 NFS/srred1kt01@WORK.LOCAL
5 12/02/13 16:14:15 HOST/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 HOST/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 HOST/srred1kt01.work.local@WORK.LOCAL
5 12/02/13 16:14:15 HOST/SRRED1KT01@WORK.LOCAL
5 12/02/13 16:14:15 HOST/SRRED1KT01@WORK.LOCAL
5 12/02/13 16:14:15 HOST/SRRED1KT01@WORK.LOCAL

The export on the NFS server looks like this:

/foo gss/krb5(ro,fsid=0,sync,insecure,no_root_squash,no_subtree_check,squash_uids=0-99)
/foo/bar gss/krb5(rw,insecure,no_subtree_check,nohide,sync,no_root_squash,squash_uids=0-99)

When I try to mount the export on the client, I see this from the mount command:

[root@srred1kt01 ~]# mount -t nfs4 -o sec=krb5 srred1nfs01:/foo /backups -vvv
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "srred1nfs01:/foo"
mount: node:  "/backups"
mount: types: "nfs4"
mount: opts:  "sec=krb5"
final mount options: 'sec=krb5'
mount: external mount: argv[0] = "/sbin/mount.nfs4"
mount: external mount: argv[1] = "srred1nfs01:/foo"
mount: external mount: argv[2] = "/backups"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sec=krb5"
mount.nfs4: timeout set for Mon Dec  2 16:26:44 2013
mount.nfs4: trying text-based options 'sec=krb5,addr=10.10.10.63,clientaddr=10.10.10.62'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting srred1nfs01:/foo

The output of "rpc.gssd -fvvv" on the client looks like this:

dir_notify_handler: sig 37 si 0x7fffaa8850b0 data 0x7fffaa884f80
dir_notify_handler: sig 37 si 0x7fffaa8850b0 data 0x7fffaa884f80
dir_notify_handler: sig 37 si 0x7fffaa8850b0 data 0x7fffaa884f80
handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt14)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt14)
process_krb5_upcall: service is '<null>'
Full hostname for 'srred1nfs01.work.local' is 'srred1nfs01.work.local'
Full hostname for 'srred1kt01.work.local' is 'srred1kt01.work.local'
No key table entry found for SRRED1KT01.WORK.LOCAL$@WORK.LOCAL while getting keytab entry for 'SRRED1KT01.WORK.LOCAL$@WORK.LOCAL       '
Success getting keytab entry for 'root/srred1kt01.work.local@WORK.LOCAL'
WARNING: Client not found in Kerberos database while getting initial ticket for principal 'root/srred1kt01.work.local@WORK.LOCAL' using keytab 'FILE:/etc/krb5.keytab'
ERROR: No credentials found for connection to server srred1nfs01.work.local
doing error downcall
dir_notify_handler: sig 37 si 0x7fffaa884b70 data 0x7fffaa884a40
dir_notify_handler: sig 37 si 0x7fffaa884b70 data 0x7fffaa884a40
dir_notify_handler: sig 37 si 0x7fffaa884b70 data 0x7fffaa884a40
dir_notify_handler: sig 37 si 0x7fffaa884b70 data 0x7fffaa884a40
dir_notify_handler: sig 37 si 0x7fffaa884b70 data 0x7fffaa884a40
dir_notify_handler: sig 37 si 0x7fffaa884b70 data 0x7fffaa884a40
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt14
Peter Loron
  • 168
  • 2
  • 8

1 Answers1

2

Not completely sure if this will help but it seems so familiar like an issue I just had to troubleshoot...

My users were experiencing random access denied messages from a CIFS share off one particular Netapp we have. It took a long while to figure out the issue because it was so intermittent. I finally narrowed it down to our 2 2012 Domain Controllers. In Server 2012 there is a new Kerberos feature introduced called "SID Compression." There are a lot of NAS devices that have issues with this because it cannot speak this new piece of Kerberos unless the vendor upgrades the code. You will receive access denied errors because Kerberos will be denied negotiating tickets from a 2012 KDC. Our issue was we only had 2 DCs running 2012 and the rest are 2008 which is the reason our issue was so intermittent since it was only denying clients that just so happen to get their tickets from those 2 2012 DCs.

Since you only have 2012 in your environment I wonder if both your Netapp and CentOS can support SID compression. OnTap 8.1.2P2 is the first release that adds this feature, anything previous will not work. The nature of how the Kerberos ticket is denied will not even allow the Netapp to fall back to NTLM because it thinks something suspicious is going on.

Here is the MS KB that deals directly with the issue and explains a fix if you cannot upgrade your Netapp/CentOS code: http://support.microsoft.com/kb/2774190. You can change the msDS-SupportedEncryptionTypes attribute of the AD computer object of your Netapp or CentOS box which will ignore SID Compression for Kerberos only for those machines thus allowing them to authenticate correctly.

Again, not sure if this is exactly the problem but it's so close I figured I had to mention it.

MethoteK
  • 21
  • 2