Kubernetes metrics-server having SSL trouble

Question

We have tried to set up metrics server in our kubernetes cluster, and it keeps failing.

I am a bit unsure where I went wrong. The cluster has been set up and upgraded using kubeadm on existing hardware. I see that during a lot of operations kubernetes tries and fails to communicate with metrics-server.

Anyone has experienced this and/or can help me finding the cause of this?

Here some output from the metric-server logs:

I0201 09:20:32.016226       1 manager.go:150] ScrapeMetrics: time: 216.595261ms, nodes: 5, pods: 49
I0201 09:20:32.016257       1 manager.go:115] ...Storing metrics...
I0201 09:20:32.016319       1 manager.go:126] ...Cycle complete
E0201 09:20:32.596639       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.596839       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (615.212µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:32.636449       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.636590       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (460.541µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
I0201 09:20:37.552609       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.552813       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtZXRyaWNzLXNlcnZlci10b2tlbi10bTRmdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMyN2VjNDAxLTI1NjMtMTFlOS04Y2M2LTFjYzFkZWVhNzdjOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTptZXRyaWNzLXNlcnZlciJ9.YF4IaGtM_IlRZ8Xzik3AEDnv6-Q4YQBamBjna_gLydhVehH4gmq_Y4y0Nrcqt4Ana9HwNcLx0jGV4GU-njUfzrb0uS9eKl2Eeh6bLTkwafKAv7cF8SwP0rBLuhIl6FDgwBU4d95MQAqOxvMdnlSquJmYOiuIT25OxD_wPJ2PYjdXbuxxSChvrLrtGwa5URbzNvN9deMWSugbz2B1knCu8YAlKPx31bUEa27YFCZIrtydRjY2E1Qzl8hkJiEuom8v_sRLTvnJyYcOU6ARWqwJT570JeubMO5_GcvnpVpmBmh8QFr8_BLTJJfiEleFNs9YmBgWIr3xDwjEBDmn5ndjrQ" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0201 09:20:37.572204       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 19 milliseconds
I0201 09:20:37.572235       1 round_trippers.go:411] Response Headers:
I0201 09:20:37.572245       1 round_trippers.go:414]     Content-Type: application/json
I0201 09:20:37.572254       1 round_trippers.go:414]     Content-Length: 260
I0201 09:20:37.572262       1 round_trippers.go:414]     Date: Fri, 01 Feb 2019 09:20:37 GMT
I0201 09:20:37.572323       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.572465       1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:37.572580       1 wrap.go:42] GET /: (20.227877ms) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.404760       1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.404908       1 wrap.go:42] GET /: (321.809µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.451089       1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.451212       1 wrap.go:42] GET /: (283.995µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
E0201 09:20:40.708131       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.708327       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (544.441µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/controller-discovery] 10.46.0.0:44210]
E0201 09:20:40.955975       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.956151       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (574.914µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:41.785405       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:41.785570       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (579.992µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:42.065074       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.065248       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (566.86µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:42.305102       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.305272       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (552.597µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]

And this from kube-apiserver logs:

I0201 09:22:14.652152       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:22:19.688846       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:22:49.751772       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:19.816917       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:49.896396       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:24:14.314774       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:24:14.317317       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:24:14 GMT] Content-Type:[application/json]]
I0201 09:24:14.317368       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:24:19.960927       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:24:50.037553       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:25:14.317811       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:25:14.320556       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:25:14 GMT] Content-Type:[application/json]]
I0201 09:25:14.320623       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:25:20.110375       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:25:50.172368       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized

score 2 · Answer 1 · answered Feb 01 '19 at 09:51

2

This issue has been reported on https://github.com/kubernetes/kubernetes/issues/69277 and further discussed on https://github.com/kubernetes/kubernetes/issues/61879. As stated, Multi-master setup the ca crt/key files should be generated outside and set in the /etc/kubernetes/pki/* folder so that Kubeadm can issue the server cert and client certs using the ca files. I hope this helps.

answered Feb 01 '19 at 09:51

Jose Antonio

76
4

This sounds a lot like it is the problem I have, but how do I fix it? – Flyhard Feb 01 '19 at 14:20

score 2 · Accepted Answer · answered Feb 01 '19 at 14:34

Following the https://github.com/kubernetes-incubator/metrics-server/issues/67, https://github.com/kubernetes-incubator/metrics-server/issues/146 and https://github.com/kubernetes-incubator/metrics-server/issues/131 you may want try use next solution:

For future readers scratching their heads: on a Kubernetes 1.13 cluster deployed with kubeadm, metrics server started working once I updated the deployment spec with the following:

 command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP

(After that, give it a few minutes before kubectl top actually has enough data to show anything, though.)

Or at least try to modify metrics-server Deployment to

    command:
    - /metrics-server
    - --kubelet-insecure-tls

Kubernetes metrics-server having SSL trouble

2 Answers2