K3S can't pull public docker hub images

Question

K3S cluster is failling to pull public docker hub images. in this instance i will user linuxserver/webtop which presents the error

Failed to pull image "linuxserver/webtop": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/linuxserver/webtop:latest": failed to copy: httpReaderSeeker: failed open: failed to do request: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/93/935ea2a17ae86691c14df443e21f3c69a6f75609894f8cfe9378b66e39c0d752/data?verify=1634025162-%2Bc4663u0w0ArAbfgBm4aFRfJUh4%3D": dial tcp: lookup production.cloudflare.docker.com: no such host

i have found lots of similar threads to this one but they are all specific to trying to pull private images which causes this problem. So although could be related i think it is something else.

I did think it may be related to rate limiting on the dockerhub site but i have tried pulling the image directly from all of the nodes individually using docker pull and it works fine so seems the problem is elsewhere.

i have tried adding my docker credentials to the deployment with imagePullSecrets: but it didn't resolve.

due to the error i feel as though it's worth mentioning that on the nodes i'm using 1.1.1.1 as the dns

If anyone has any idea what could be the problem would be appreciated.

also seems to be intermittent for example here.

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  7m2s                  default-scheduler  Successfully assigned default/dnstest-l6dh2 to node-3
  Warning  Failed     7m1s                  kubelet            Failed to pull image "busybox:1.28": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:1.28": failed to resolve reference "docker.io/library/busybox:1.28": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/1.28": dial tcp: lookup registry-1.docker.io: no such host
  Warning  Failed     7m1s                  kubelet            Error: ErrImagePull
  Normal   BackOff    7m1s                  kubelet            Back-off pulling image "busybox:1.28"
  Warning  Failed     7m1s                  kubelet            Error: ImagePullBackOff
  Normal   Pulling    6m49s (x2 over 7m2s)  kubelet            Pulling image "busybox:1.28"
  Normal   Pulled     6m47s                 kubelet            Successfully pulled image "busybox:1.28" in 1.3937859s
  Normal   Created    6m47s                 kubelet            Created container alpine
  Normal   Started    6m47s                 kubelet            Started container alpine

Your node can't resolve `production.cloudflare.docker.com`. You have a DNS issue. — Gerald Schneider, Oct 12 '21 at 07:17
yes but on the node it resolves it fine as i can pull the image so where do i need to change the dns? — locose, Oct 12 '21 at 07:29

score 0 · Answer 1 · answered Jul 24 '22 at 23:25

0

I was having this issue too. My problem was I had multiple default routes.

I had a whole write up about this over here,

https://devops.stackexchange.com/a/16244/18965

If you can check this and confirm this is what your problem is too, I'd be very thankful. You didn't give me much to go on, but this lines up with 100% of what I saw. The problem took me a week to solve.

answered Jul 24 '22 at 23:25

Evan Carroll

2,245
10
34
50

Thanks Evan Sorry about the delay. kind of changed jobs and the env no longer exists but i'll read it and let you know anyway – locose Aug 09 '22 at 08:04
Thanks Evan looks like a good solution. I no longer have the environment and wasn't using terraform but certainly seems like something that might of solved the issue. – locose Aug 09 '22 at 08:08

K3S can't pull public docker hub images

1 Answers1