We have tried to set up metrics server in our kubernetes cluster, and it keeps failing.
I am a bit unsure where I went wrong. The cluster has been set up and upgraded using kubeadm on existing hardware. I see that during a lot of operations kubernetes tries and fails to communicate with metrics-server.
Anyone has experienced this and/or can help me finding the cause of this?
Here some output from the metric-server logs:
I0201 09:20:32.016226 1 manager.go:150] ScrapeMetrics: time: 216.595261ms, nodes: 5, pods: 49
I0201 09:20:32.016257 1 manager.go:115] ...Storing metrics...
I0201 09:20:32.016319 1 manager.go:126] ...Cycle complete
E0201 09:20:32.596639 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.596839 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (615.212µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:32.636449 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.636590 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (460.541µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
I0201 09:20:37.552609 1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.552813 1 round_trippers.go:386] curl -k -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtZXRyaWNzLXNlcnZlci10b2tlbi10bTRmdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMyN2VjNDAxLTI1NjMtMTFlOS04Y2M2LTFjYzFkZWVhNzdjOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTptZXRyaWNzLXNlcnZlciJ9.YF4IaGtM_IlRZ8Xzik3AEDnv6-Q4YQBamBjna_gLydhVehH4gmq_Y4y0Nrcqt4Ana9HwNcLx0jGV4GU-njUfzrb0uS9eKl2Eeh6bLTkwafKAv7cF8SwP0rBLuhIl6FDgwBU4d95MQAqOxvMdnlSquJmYOiuIT25OxD_wPJ2PYjdXbuxxSChvrLrtGwa5URbzNvN9deMWSugbz2B1knCu8YAlKPx31bUEa27YFCZIrtydRjY2E1Qzl8hkJiEuom8v_sRLTvnJyYcOU6ARWqwJT570JeubMO5_GcvnpVpmBmh8QFr8_BLTJJfiEleFNs9YmBgWIr3xDwjEBDmn5ndjrQ" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0201 09:20:37.572204 1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 19 milliseconds
I0201 09:20:37.572235 1 round_trippers.go:411] Response Headers:
I0201 09:20:37.572245 1 round_trippers.go:414] Content-Type: application/json
I0201 09:20:37.572254 1 round_trippers.go:414] Content-Length: 260
I0201 09:20:37.572262 1 round_trippers.go:414] Date: Fri, 01 Feb 2019 09:20:37 GMT
I0201 09:20:37.572323 1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.572465 1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:37.572580 1 wrap.go:42] GET /: (20.227877ms) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.404760 1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.404908 1 wrap.go:42] GET /: (321.809µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.451089 1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.451212 1 wrap.go:42] GET /: (283.995µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
E0201 09:20:40.708131 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.708327 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (544.441µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/controller-discovery] 10.46.0.0:44210]
E0201 09:20:40.955975 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.956151 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (574.914µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:41.785405 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:41.785570 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (579.992µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:42.065074 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.065248 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (566.86µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:42.305102 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.305272 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (552.597µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
And this from kube-apiserver logs:
I0201 09:22:14.652152 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:22:19.688846 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:22:49.751772 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:19.816917 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:49.896396 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:24:14.314774 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:24:14.317317 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:24:14 GMT] Content-Type:[application/json]]
I0201 09:24:14.317368 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:24:19.960927 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:24:50.037553 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:25:14.317811 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:25:14.320556 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:25:14 GMT] Content-Type:[application/json]]
I0201 09:25:14.320623 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:25:20.110375 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:25:50.172368 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized