Lately my GCP VM of multiple GPUs throws the following error when I try to run my container:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container ini
t caused: Running hook #1:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: timed out: unknown
I also noticed, that executing nvidia-smi
takes more than 30 seconds most of the time.
Specs:
- base image:
nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
- nvidia driver: 450.102.04
- zone: europe-west1-b
I've been using this setup for months and never noticed anything alike.