Since the failure of quay.io last week everyone in the company recognized how important is the service. And this raised this question on my head:
If this happens again, how can my infrastructure be prepared to use a secondary docker container registry in case the first one is unavailable?
As of now, the docker engine has a --registry-mirror option, but that only works for configuring a mirror of Docker hub. There have been some open issues looking at whether or how that can be adjusted to allow mirroring other registries, but there hasn't been enough traction to make that change. If your upstream registry is docker hub, using this flag to point to a local cache or mirror is very useful since requests to hub automatically try this mirror first, and on failure will then try the upstream registry. This means you don't need to adjust your image names or worry about a local outage as much.
Containerd has a similar option, and it supports any upstream registry, so I believe this is what we will go to eventually. However, if you are using docker, even with containerd under the covers, I believe the docker engine is still doing the image push and pull.
Without the ability to configure this in containerd, I've seen a few options for caching images from other registries:
Configure an HTTP caching proxy like squid. You do want to ensure this periodically refreshes the mutable responses like image manifests. The layers themselves should remain unchanged so you can cache them for a long time.
Pull directly from another mirror or caching registry.
Use some DNS/TLS hijacking to send requests to your cache or mirror.
Configuring a pull through caching registry is pretty easy:
docker run -p 5000:5000 -e REGISTRY_PROXY_REMOTEURL=<upstream-url> registry:2
However, I tend to run a mirror instead because the pull through cache is configured with a 7 day expiration of entries. So if the outage happens and your entries have recently expired from your pull through cache, you'll be impacted by that outage.
Running a mirror involves running a registry, same as the pull through cache does, and then replicating those images you want to mirror. Some registries, Harbor comes to mind, offer mirroring as a feature. You specify which remote registries to mirror to or from. I personally do the mirroring with a shell script since that allows me to backup old images (most mirrors will replace the tag if it was replaced upstream, leaving you no revert option). I then include that script in my CI/CD workflows to run at the appropriate time, e.g. at the start of a sprint, or every Monday morning. An example of this mirroring script is in my presentation repo.
Once you have the image available in a local registry, you'll need to configure docker to pull from that registry. Without the --registry-mirror option, you either need to adjust your image names, or configure DNS/TLS on your network/docker hosts to point requests to your local registry instead of the upstream one. The latter option looks attractive since it allows you to run your docker commands without any changes, image builds, compose stacks, helm charts, etc all work as if they were talking to the remote registry. However it's a bit more error prone managing your own TLS certificates and a self signed CA that all hosts on your network need to be configured to trust. So the easy option for me has been to adjust the registry name. In my Dockerfile I'll have something like:
ARG REGISTRY=docker.io
FROM ${REGISTRY}/library/alpine:3.9
That allows this image to be built by others outside of the network. And then in my build commands on my network, I'll override the arg with my local mirror:
docker build --build-arg REGISTRY=local-mirror:5000 .
I do the same in docker-compose.yml files with a variable:
image: ${REGISTRY:-docker.io}/my/repo:and-tag
And then deploy that locally with something like:
REGISTRY=local-mirror:5000 docker-compose up -d
I cover this in my DockerCon presentation. Slides are posted to my github at: https://github.com/sudo-bmitch/presentations/tree/master/registry
The presentation is online at: https://www.youtube.com/watch?v=Bm7g0saAC9k
