0

We have a Kubernetes cluster running on Google Container Engine. The Container clusters section of the Google cloud console shows that the cluster api version is 1.06.

I tried to do a rolling update to upgrade to the latest api version (1.1.1 is currently most up to date available).

I ran: gcloud container clusters upgrade my-cluster-name and received an error (identifiers removed)

(gcloud.container.clusters.upgrade) Operation [https://container.googleapis.com/v1/projects/' status: StatusValueValuesEnum(DONE, 3) statusMessage: u'Error upgrading nodes: Cluster nodes did not update in time. Rolling update with ID "1" last had status: "2/3 instances updated; Errors: CONDITION_NOT_MET (Too many (1 > 0) instances failed.)".' targetLink: u'https://container.googleapis.com/v1/projects/' zone: u'1'>] finished with error: Error upgrading nodes: Cluster nodes did not update in time. Rolling update with ID "1" last had status: "2/3 instances updated; Errors: CONDITION_NOT_MET (Too many (1 > 0) instances failed.)".

I tried running the command again and received this error:

(gcloud.container.clusters.upgrade) Operation [https://container.googleapis.com/v1/projects/1' status: StatusValueValuesEnum(DONE, 3) statusMessage: u'Error upgrading nodes: A problem occurred during the rolling update.' targetLink: u'https://container.googleapis.com/v1/projects/1' zone: u'1'>] finished with error: Error upgrading nodes: A problem occurred during the rolling update.

any ideas?

Andreas Veithen
  • 386
  • 1
  • 4
toby
  • 135
  • 1
  • 5
  • 1
    The master node for your Container Engine Cluster is hosted by Google. As such, the version of Kubernetes on master is automatically updated. Can you check if your master node is updated to the newer version before updating other nodes in your cluster. You can refer to this link for more information (https://cloud.google.com/container-engine/docs/clusters/upgrade) – Faizan Nov 13 '15 at 18:07

1 Answers1

2

Container Engine doesn't allow nodes to be upgraded to a version higher than the master, because Kubernetes currently only ensures that version-skewed clusters work properly with the master at a newer version than the nodes, but not the other way around. There is ongoing discussion about whether the opposite version skew should be fully supported as well.

We haven't yet started upgrading existing GKE masters to 1.1.1, but we'll start doing so soon! In the meantime, new GKE clusters are being created at version 1.1.1 as of this week, so if you have a particular feature you want to try out that can't wait, creating a new cluster should get you access to it.

Alex Robinson
  • 311
  • 1
  • 4
  • 1
    If you really want your existing clusters upgraded as soon as possible, you can also reach out to Google's cloud support and file a request. – Robert Bailey Nov 13 '15 at 21:42
  • Just a note 'gcloud container clusters describe' command will show u the version that the master is running, and what u can upgrade to. – chrislovecnm Nov 22 '15 at 05:52