1

I have executed the following command on a on this point in time three times mirrored regional cluster:

gcloud container clusters update proto-cluster-ha-1 --zone europe-north1 --node-locations europe-north1-a

Before the execution the cluster was mirrored across zones europe-north1-a / europe-north1-b / europe-north1-c.

After the execution the cluster only operates in the single zone europe-north1-a.

The problem:

Since the update my volume mountings are no longer able to mount because of:

2 node(s) had no available volume zone

although the PVC is displayed as BOUND:

csi-pvc-mounting         Bound

If I delete a mounting a recreate it, everything works fine. But how can I recover such a volume mounting without destroying it?

I already compared the connected yaml of a newly created mounting with an old one that does not want to bound but I can detect no differences..

tech34841916559242495
  • 11
  • 2
  • I'm unable to reproduce this issue on my end. Could you please confirm if you are still experiencing this issue? Also could let me know what this issue occurred, date and timezone so I can validate if any issue was reported in this region at that time. – Sunny J Mar 05 '19 at 04:36
  • you should be able to take a snapshot, recreate the volume from the snapshot in a different zone, and then create those pv/pvc resources using the generated id – Patrick Lee Scott Sep 24 '19 at 21:08

1 Answers1

1

This question is bit old, however if someone would still have similar issue, documentation was updated well regarding this topic.

I have tried to reproduce this behavior, however output of using this command is working as designed.

When you used

$ gcloud container clusters update proto-cluster-ha-1 --zone europe-north1 --node-locations europe-north1-a

You have updated cluster proto-cluster-ha-1 to use only one zone europe-north1-a.

It's well described in --node locations parameter.

The set of zones in which the specified node footprint should be replicated. All zones must be in the same region as the cluster's master(s), specified by the --zone or --region flag. Additionally, for zonal clusters, --node-locations must contain the cluster's primary zone. If not specified, all nodes will be in the cluster's primary zone (for zonal clusters) or spread across three randomly chosen zones within the cluster's region (for regional clusters).

If you would like to update configuration in more than one zone, you would need to specify it. Example below:

$ kubectl describe node | grep zone
                    failure-domain.beta.kubernetes.io/zone=us-central1-c
                    failure-domain.beta.kubernetes.io/zone=us-central1-b
                    failure-domain.beta.kubernetes.io/zone=us-central1-a
                    
$ gcloud container clusters update regio --zone us-central1 --node-locations us-central1-a,us-central1-b
Updating regio...done.
Updated [https://container.googleapis.com/v1/projects/XXXXXXX/zones/us-central1/clusters/regio].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-central1/regio?project=XXXXXXXX
$ kubectl describe node | grep zone
                    failure-domain.beta.kubernetes.io/zone=us-central1-b
                    failure-domain.beta.kubernetes.io/zone=us-central1-a

Regarding volume, you did not mention how you created this Volume, what Reclaim Policy was used.

PersistentVolumes can have various reclaim policies, including Retain, Recycle, and Delete. For dynamically provisioned PersistentVolumes, the default reclaim policy is Delete. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume contains precious data. In that case, it is more appropriate to use the "Retain" policy. With the Retain policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume` is not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.

**Depends on that you could: **

Reclaim

When a user is done with their volume, they can delete the PVC objects from the API that allows reclamation of the resource. The reclaim policy for a PersistentVolume tells the cluster what to do with the volume after it has been released of its claim. Currently, volumes can either be Retained, Recycled, or Deleted.

or

Retain

The Retain reclaim policy allows for manual reclamation of the resource. When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume. An administrator can manually reclaim the volume with the following steps.

Currently you can create Regional Persistent Disks to avoid this kind of situation.

PjoterS
  • 615
  • 3
  • 11