I'm trying to mount a directory into a client server using kerberos authentication.
If I create a keytab file using using kadmin in the server, I cannot get authenticated when I mount the directory.
sudo kadmin -p root/admin -w $KERBEROS_PASSWORD ktadd nfs/kbserver.example.com
sudo kadmin -p root/admin -w $KERBEROS_PASSWORD ktadd host/kbserver.example.com
sudo kadmin -p root/admin -w $KERBEROS_PASSWORD ktadd nfs/kube-node-0.example.com
sudo kadmin -p root/admin -w $KERBEROS_PASSWORD ktadd host/kube-node-0.example.com
udo kdestroy -A
sudo kinit -k -t /etc/krb5.keytab
sudo systemctl restart nfs-secure
sudo mount -t nfs4 -o sec=krb5 kbserver.example.com:/ /home/ec2-user/nfs-test
The result of that is:
kbserver.example.com:/ /home/ec2-user/nfs-test -v
mount.nfs4: timeout set for Fri Sep 7 23:13:53 2018
mount.nfs4: trying text-based options 'sec=krb5,vers=4.1,addr=10.1.5.28,clientaddr=10.1.1.248'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5,vers=4.0,addr=10.1.5.28,clientaddr=10.1.1.248'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting kbserver.example.com:/
If I, on the other hand, do the following on the server:
[ec2-user@kbserver ~]$ sudo kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local: ktadd host/kbserver.example.com
Entry for principal host/kbserver.example.com with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
...
kadmin.local: ktadd nfs/kbserver.example.com
Entry for principal nfs/kbserver.example.com with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
...
kadmin.local: ktadd host/kube-node-0.example.com
Entry for principal host/kube-node-0.example.com with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
...
kadmin.local: ktadd nfs/kube-node-0.example.com
Entry for principal nfs/kube-node-0.example.com with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
...
kadmin.local: quit
sudo cat /etc/krb5.keytab | base64 -w0
And then do the following in the client, then the mount works:
echo $BASE_64_ENCODED | base64 -d | sudo tee /etc/krb5.keytab
sudo kdestroy -A && sudo kinit -k -t /etc/krb5.keytab && sudo systemctl restart nfs-secure
sudo mount -t nfs4 -o sec=krb5 kbserver.example.com:/ /home/ec2-user/nfs-test
My journalctl logs say the following:
Sep 12 18:03:55 kube-node-0.example.com polkitd[603]: Unregistered Authentication Agent for unix-process:8510:15795400 (system bus name :1.302, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Sep 12 18:03:59 kube-node-0.example.com sudo[8676]: ec2-user : TTY=pts/2 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/mount -t nfs4 -o sec=krb5 kbserver.example.com:/ /home/ec2-user/nfs-test
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]:
handle_gssd_upcall: 'mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2 ' (nfs/clnt20)
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: krb5_use_machine_creds: uid 0 tgtname (null)
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: Full hostname for 'kbserver.example.com' is 'kbserver.example.com'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: Full hostname for 'kube-node-0.example.com' is 'kube-node-0.example.com'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: No key table entry found for kube-node-0$@EXAMPLE.COM while getting keytab entry for 'kube-node-0$@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: No key table entry found for KUBE-NODE-0$@EXAMPLE.COM while getting keytab entry for 'KUBE-NODE-0$@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: No key table entry found for root/kube-node-0.example.com@EXAMPLE.COM while getting keytab entry for 'root/kube-node-0.example.com@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: Success getting keytab entry for 'nfs/kube-node-0.example.com@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: gssd_get_single_krb5_cred: principal 'nfs/kube-node-0.example.com@EXAMPLE.COM' ccache:'FILE:/tmp/krb5ccmachine_EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1536861839
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: creating tcp client for server kbserver.example.com
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: creating context with server nfs@kbserver.example.com
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: doing downcall: lifetime_rec=86400 acceptor=nfs@kbserver.example.com
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]:
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt20)
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: krb5_use_machine_creds: uid 0 tgtname (null)
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: Full hostname for 'kbserver.example.com' is 'kbserver.example.com'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: Full hostname for 'kube-node-0.example.com' is 'kube-node-0.example.com'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: No key table entry found for kube-node-0$@EXAMPLE.COM while getting keytab entry for 'kube-node-0$@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: No key table entry found for KUBE-NODE-0$@EXAMPLE.COM while getting keytab entry for 'KUBE-NODE-0$@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: No key table entry found for root/kube-node-0.example.com@EXAMPLE.COM while getting keytab entry for 'root/kube-node-0.example.com@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: Success getting keytab entry for 'nfs/kube-node-0.example.com@EXAMPLE.COM'
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1536861839
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1536861839
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: creating tcp client for server kbserver.example.com
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: creating context with server nfs@kbserver.example.com
Sep 12 18:03:59 kube-node-0.example.com rpc.gssd[8534]: doing downcall: lifetime_rec=86400 acceptor=nfs@kbserver.example.com
Sep 12 18:0
I've checked that my confis and hosts files are identical and have the correct hosts:
[ec2-user@kube-node-0 ~]$ sudo md5sum /etc/krb5.conf
808b4fd2b3c97a89d1a13a464afad6f0 /etc/krb5.conf
[ec2-user@kube-node-0 ~]$ sudo md5sum /etc/hosts
bf563e1b1288cb87f7152658c926215f /etc/hosts
Server:
[ec2-user@kbserver ~]$ sudo md5sum /etc/krb5.conf
808b4fd2b3c97a89d1a13a464afad6f0 /etc/krb5.conf
[ec2-user@kbserver ~]$ sudo md5sum /etc/hosts
bf563e1b1288cb87f7152658c926215f /etc/hosts
My only hypothesis so far is that where you create they keytab file IS significant, despite the fact that they are using the same principal, but not sure what that would matter.
