1

Running kubernetes v1.10.0 amd kube-dns using the gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8 container

When the kube-dns container starts it says this in the log:

I0610 06:47:06.051414       1 round_trippers.go:398] curl -k -v -XGET  -H "User-Agent: kube-dns/1.14.10 (linux/amd64)" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp... rest of barer token" -H "Accept: application/vnd.kubernetes.protobuf, */*" https://172.17.0.1:443/api/v1/services?resourceVersion=0
`E0610 06:47:05.058513       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:189: Failed to list *v1.Endpoints: Get https://172.17.0.1:443/api/v1/endpoints?resourceVersion=0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.16.23.40")`

10.16.23.40 is the "real" address of my master, 172.17.0.1 is the service address of the master.

I have a self signed certificate on the apiserver, which understandably throws the usual verification errors if accessed with curl (from the command line of anything), without -k. The output from kube-dns however seems to imply that -k is used, although I can find no evidence of curl actually existing on the container, which leads me to believe that the binary curl is not actually being called, but the curl command is just being output to "help". The existence of toCurl in round_trippers.go seems to back this theory up.

Looking at the output, yes, the certificate is self-signed, and therefore "signed by an unknown authority" makes a certain amount of sense, but if its making the call with -k (or whatever its equivalent is within the code) Then that should not be an issue.

Here are some relevant lines in the certificate:

Subject: C=UK, ST=Blah, L=Blah, O=system:nodes, OU=Blah, CN=10.16.23.40

And 

X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, DNS:cluster.local, DNS:k8s.blah.net, IP Address:10.16.23.40, IP Address:172.17.0.1

Naturally everything else that needs to access the apiserver (e.g. kubectl) executes without a problem and without any kind of verification error.

Any pointers on how to get kube-dns talking to the apiserver would be most appreciated.

Thanks In advance

GeoSword
  • 1,647
  • 12
  • 16
  • Please consider that kube-dns use two independent sets of x.509 certificates. First is used with communication between nodes and API server and second is used to sign internal Kubernetes zones changes. As I see, you have a working connection from kubectl to API server. Your issue indicates at a possible certificate mismatch with kube-dns zone signing cert. Please verify if the certificates are valid with the corresponding CA cert from Kubernetes API Server and zone signing certificates with their kube-dns CA cert. – d0bry Jun 11 '18 at 17:00

1 Answers1

2

I Found the problem was actually in the controller-manager. I had specified --master= on the command line which was overriding the value in the controller-manager config file. Removed that, restarted controller-manager kube-dns functions as expected. The verification error while apparently talking to the api server was somewhat of a red-herring.

GeoSword
  • 1,647
  • 12
  • 16