I hope this is the right place to ask, I've followed all the support links from the GCP console which have led me here!
In the past I would frequently keep my GKE k8s master and nodes up to date using the GCP console. This all worked seamlessly... until now. And I have no idea why. I've found some GKE Stackdriver logs but nothing seems to be obvious as to why this is failing. I can upgrade the master with no problem, but when presented with the option to upgrades the nodes (there is only one node in the cluster) the upgrade starts, then seconds later it stops. The node remains on the same version and I'm again notified that an upgrade is available.
I also notice that I'm no longer to exec
to running Pods. The error is:
Error from server: error dialing backend: No SSH tunnels currently open. Were the targets able to accept an ssh-key for user "gke-<random-digits>"?
I'm not sure if this is related.
I've read through this: https://cloud.google.com/kubernetes-engine/docs/troubleshooting#kubect_commands_hang but it's not helped.
I tried resetting the node but that had no impact either. Well, the only effect I saw was that one of my containers could no longer mount a volume due to I/O errors!
Any ideas where to start?!
Edit
Current Master version: 1.11.6-gke.0
Current Node version: 1.10.7-gke.2
Last operation from: gcloud beta container operations describe
endTime: '2019-01-15T23:14:21.936649805Z'
name: operation-1547594061714-c2750b4c
operationType: UPDATE_CLUSTER
selfLink: https://container.googleapis.com/v1beta1/projects/348462677705/zones/europe-west2-a/operations/operation-1547594061714-c2750b4c
startTime: '2019-01-15T23:14:21.714859214Z'
status: DONE
targetLink: https://container.googleapis.com/v1beta1/projects/348462677705/zones/europe-west2-a/clusters/clubbed-cluster
zone: europe-west2-a
The above UPDATE_CLUSTER
operation occurs continually every minute of every day!