When adding new pool nodes with GPU Google Kubernetes Engine can't schedule newly created pods that demand GPU on these new nodes, should be automatic but not for GPU resources I guess, new pods stays in 'pending' state forever, how to fix that?
EDIT: Here is the deployment yaml file, I aim not to bind deployment to a specific node:
---
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SldDeployment
metadata:
labels:
app: sld
name: trs-sld
namespace: trs
spec:
annotations:
project_name: Trs
deployment_version: v1.0
seldon.io/rest-connect-retries: '5'
seldon.io/grpc-connect-retries: '5'
seldon.io/istio-retries: '10'
seldon.io/istio-retries-timeout: '12'
name: trs
predictors:
- componentSpecs:
- spec:
containers:
- image: eu.gcr.io/trs-141513/trs-native:latest
imagePullPolicy: Always
name: classifier
resources:
limits:
nvidia.com/gpu: 2
volumeMounts:
- mountPath: /etc/google_storage/creds
name: service-account-creds
readOnly: true
volumes:
- name: service-account-creds
secret:
secretName: service-account-creds
terminationGracePeriodSeconds: 20
graph:
children: []
name: classifier
endpoint:
type: REST
type: MODEL
name: model
replicas: 1
annotations:
predictor_version: v1.0
---