Hosting docker images on multiple registries but reference them the same way e.g. in a k8s deployment

Question

Since I started with docker there is one question that follows me and I could not get a good answer so far.

Let's assume I have a .yaml file specifying a k8s deployment. We are building our docker images on a local build server and they are getting pushed to a local registry e.g. registry.mycompany.com which is only accesiable in our local company network. So in my .yaml I will reference that image with "image: registry.mycompany.com/myrepo/myimage:1.0".

Now when I deploy that on a local k8s cluster it works correctly as the domain name gets resolved and the image can be pulled. Now I want to deploy the same e.g. in any Cloud. Well, it wont work as the registry is not availiable. Same goes if I want to deploy the same .yaml at one of my customers places where they also for a good reason can not access my local registry.

How is this kind of thing solved usually? Right now, I setup another registry either at my customers or in the cloud account, retag all my images and push them to that registry and also change all the image references in the .yamls when I deploy at a customers place or in the cloud. Therefore having to maintain a lot of sets of files that are 99% the same. That is not how it should work right?

My "wish" would be that I can specify just the image name and tag "myimage:1.0" and the k8s cluster just checks all availiable registries if he can find the image and pulls it. But I am also aware that there is no "list of well-known-registries", you either specify the registry in the image name or leave it out and docker hub will be queried. However it would be very cool if I can specify just

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: MyDeployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: myimage
    spec:
      containers:
        - name: myimage
          image: 'myimage:1.0'
          ports: []

and a list of registries:

registries:
    - registry.mycompany.com
    - registry.cloud.com
    - registry.localcustomer.com

But there is nothing like that, right?

The only other solution I can think of, is to build everything into helm charts, make the first part of the image reference a variable and pass the registry as a variable to the chart.

Any ideas and infos are welcome. Thanks in advance :)

Hello. As you said, the Helm chart version should be most convenient solution. Have you checked [Kustomize](https://blog.stack-labs.com/code/kustomize-101/) by any chance as an alternative solution? — Dawid Kruk, Nov 24 '20 at 18:28

score 2 · Accepted Answer · answered Nov 25 '20 at 12:29

There are some workarounds to achieve the result like you've described in your question:

Helm Chart
Kustomize
Registry

Helm Chart

As you've said in the question:

The only other solution I can think of, is to build everything into helm charts, make the first part of the image reference a variable and pass the registry as a variable to the chart.

This is subjectively most convenient way to specify the repositories for Deployment.

The example of it could like below:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu
spec:
  selector:
    matchLabels:
      app: ubuntu
  replicas: 1
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
      - name: "{{ .Values.image.repo }}{{ .Values.image.name }}:{{ .Values.image.tag }}"  
        image: 
        command:
        - sleep 
        - "infinity"

Values inside of a template could be provided either by:

values.yaml file:

image: 
  repo: gcr.io/ # <-- only an example
  name: ubuntu
  tag: latest

$ helm command like:

$ helm install ubuntu . --set=image.repo=eu.gcr.io/

A tip!

Specifying --set parameter with already existing value in values.yaml will override it (in this example)!

Kustomize

You can also use kustomize to "render" the Deployment with their respective repositories and images.

There is quite extensive explanation on how it can be done in below blog post:

Blog.stack-labs.com: Code: Kustomize 101

Example of such solution (by following above link) could be boiled down to:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - name: app
        image: foo/bar:latest
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

custom-image.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  template:
    spec:
      containers:
        - name: app # IMPORTANT 
          image: eu.gcr.io/ubuntu:latest

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

patchesStrategicMerge:
- custom-image.yaml

After running above example with:

kustomize build

You should see changes in image:

from: image: foo/bar:latest
to: image: eu.gcr.io/ubuntu:latest

You could also take a look on this StackOverflow answer:

Stackoverflow.com: Questions: Kustomize add imagepullsecrets to all deployments

Registry

You can also take a look on registry project such as Harbor.

You can read on their github page:

Harbor is an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management.

Features

Cloud native registry: With support for both container images and Helm charts, Harbor serves as registry for cloud native environments like container runtimes and orchestration platforms.

Role based access control: Users access different repositories through 'projects' and a user can have different permission for images or Helm charts under a project.

Policy based replication: Images and charts can be replicated (synchronized) between multiple registry instances based on policies with using filters (repository, tag and label). Harbor automatically retries a replication if it encounters any errors. This can be used to assist loadbalancing, achieve high availabiliy, and faciliate multi-datacenter deployments in hybrid and multi-cloud scenarios.

...

-- Github.com: Goharbor: Harbor

With it you could:

Create repositories that could be used only by authenticated and authorized accounts.
Use it for replication purposes between locations.

Adding to that you could also look on Docker Registry proxy like the one below:

Hub.docker.com: Rpardini: Docker registry proxy

Additional resources:

Thanks a lot for that answer :) We already use harbor. However, with each registry at each location I would still have another name. E.g. I would have to pull registry.mycompany.com/myrepo/myimage:1.0, registry.cloud.com/myrepo/myimage:1.0 and registry.localcustomer.com/myrepo/myimage:1.0 depending on which location I am resulting in my original issue. Right now it looks for me the helm chart solution is the way to go. — Mangosniper, Nov 25 '20 at 20:46

Hosting docker images on multiple registries but reference them the same way e.g. in a k8s deployment

1 Answers1

Helm Chart

Kustomize

Registry

Features