1

I am getting image pull outage due to reaching the image pull count limit from docker hub, see "Events" section of the output of pod describe below.

It seems a pod in the kubernetes cluster I have setup is not finding the image on docker hub and hence failing the pod construction phase. Kubernetes then tries to construct a new pod for with for the same image and attempts to pull the same image once more.

In Kubernetes, is there a way to limit the number of times images are pulled from an image repository.

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  6m3s                 default-scheduler  Successfully assigned calico-system/calico-node-rq98n to centos7-03-10
  Warning  Failed     5m15s                kubelet            Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.24.0": rpc error: code = Unknown desc = initializing source docker://calico/pod2daemon-flexvol:v3.24.0: reading manifest v3.24.0 in docker.io/calico/pod2daemon-flexvol: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   Pulling    108s (x4 over 6m3s)  kubelet            Pulling image "docker.io/calico/pod2daemon-flexvol:v3.24.0"
  Warning  Failed     62s (x4 over 5m15s)  kubelet            Error: ErrImagePull
  Warning  Failed     62s (x3 over 3m53s)  kubelet            Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.24.0": rpc error: code = Unknown desc = reading manifest v3.24.0 in docker.io/calico/pod2daemon-flexvol: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   BackOff    21s (x7 over 5m15s)  kubelet            Back-off pulling image "docker.io/calico/pod2daemon-flexvol:v3.24.0"
  Warning  Failed     21s (x7 over 5m15s)  kubelet            Error: ImagePullBackOff
Allan K
  • 111
  • 2

1 Answers1

0

1 . On November 20, 2020, rate limits anonymous and free authenticated use of Docker Hub went into effect. Anonymous and Free Docker Hub users are limited to 100 and 200 container image pull requests per six hours. You can refer to the download rate limit for more detailed information.

As stated in the documentation :

The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect. Image requests exceeding these limits will be denied until the six hour window elapses.

So as a workaround you can either:

  1. Reduce your pull rate.
  2. Upgrade your membership.
  3. Setup your own docker proxy to cache containers locally

To overcome docker hub pull rate limit refer to the documentation and also refer to the stackpost.

2 . Another workaround is to pull the image locally once, push it to your local docker repository and then update your image properties to point to your local repository.

You have to pull the images locally using your credentials and push it to your local (internally hosted) docker repository. Once pushed, update the deployment.yaml file with updated image link.

Note: Also, you can use the Registry Mirror feature to the number of image pull requests generated against DockerHub. When the mirror is configured and GitLab Runner instructs Docker to pull images, Docker will check the mirror first; if it's the first time the image is being pulled, a connection will be made to DockerHub. Subsequent pulls of that image will then use your mirror instead of connecting to DockerHub. Refer Registry as a pull through cache for more information.

Jyothi Kiranmayi
  • 175
  • 5
  • Thank you @Jyothi Kiranmayi for your answer. Option 1 "Reduce your pull rate" is what I am seeking a cluster wide implementation of. Option 2 "Upgrade your membership" may not solve the issue the cluster will still max out the increased limits offered by the membership upgrade and also it does not prevent the lose of resources (such as bandwidth) consumed by the perpetual pull action of the images which cannot be pulled. Option 3 Does does not prevent repeated pulls of the "unpullable image" . On the plus side the "pullable" images will still be pulled and it does not result in their outage. – Allan K Aug 30 '22 at 18:21