We experienced a major outage with our production systems today. Without any user interaction most of the kubernetes pods went down with a "ImagePullBackOff" error message.
We had to manually restart builds and repush all images. I verified that in the container registry the referred images exist - this is the case, there was no change done. Even with images being displayed as avaliable, we had to repush them to get rid of the error.
What happened there?!
EDIT
Docker is unable to find the image.
$ docker pull eu.gcr.io/seepex-cs/scs-grafana
Using default tag: latest
latest: Pulling from seepex-cs/scs-grafana
9d48c3bd43c5: Already exists
4842084dac50: Already exists
7cbaa73b9ead: Already exists
9a7207a7a1b5: Already exists
6bb6df97bf66: Already exists
e9c24addd21e: Already exists
21ae065ef1d6: Already exists
error pulling image configuration: unknown blob
Image is listed in google container registry: