We have an horizontal pod auto-scaler (HPA) installed on gke cluster, most of the time the auto-scaler works perfectly fine. but from time to time (mostly on our customer rush hours) the auto scaler is getting error code 503 from stack driver.
here are the errors we encounter:
Failed request to stackdriver api: googleapi: Error 503: The service is currently unavailable., backendError
and
"apiserver received an error that is not an metav1.Status: &googleapi.Error{Code:503, Message:"The service is currently unavailable.", Body:"{\n \"error\": {\n \"code\": 503,\n \"message\": \"The service is currently unavailable.\",\n \"errors\": [\n {\n \"message\": \"The service is currently unavailable.\",\n \"domain\": \"global\",\n \"reason\": \"backendError\"\n }\n ],\n \"status\": \"UNAVAILABLE\"\n }\n}\n", Header:http.Header(nil), Errors:[]googleapi.ErrorItem{googleapi.ErrorItem{Reason:"backendError", Message:"The service is currently unavailable."}}}"
Now I'm a bit puzzled here; google is encouraging to use the stackdriver as a source of HPA's (https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling), but if it isn't 100% available or fault tolerant - the cluster is just broken as the pods are not scaling up and the resources are being exhausted..
Anyone know how to work around here ?