Docker containers can't resolve DNS on Ubuntu 14.04 Desktop Host

Question

I'm running into a problem with my Docker containers on Ubuntu 14.04 LTS. Docker worked fine for two days, and then suddenly I lost all network connectivity inside my containers. The error output below initially lead me to believe it was because apt-get is trying to resolve the DNS via IPv6.

I disabled IPv6 on my host machine and still, removed all images, pulled base ubuntu, and still ran into the problem.

I changed my /etc/resolve.conf nameservers from my local DNS server to Google's public DNS servers (8.8.8.8 and 8.8.4.4) and still have no luck. I also set the DNS to Google in the DOCKER_OPTS of /etc/default/docker and restarted docker.

I also tried pulling coreos, and yum could not resolve DNS either.

It's weird because while DNS does not work, I still get a response when I ping the same update servers that apt-get can't resolve.

I'm not behind a proxy, I'm on a very standard local network, and this version of Ubuntu is up to date and fresh (I installed two days ago to be closer to docker).

I've thoroughly researched this through other posts on stackoverflow and github issues, but haven't found any resolution. I'm out of ideas as to how to solve this problem, can anyone help?

Error Message

➜  arthouse git:(docker) ✗ docker build --no-cache .
Sending build context to Docker daemon 51.03 MB
Sending build context to Docker daemon 
Step 0 : FROM ubuntu:14.04
 ---> 5506de2b643b
Step 1 : RUN apt-get update
 ---> Running in 845ae6abd1e0
Err http://archive.ubuntu.com trusty InRelease
Err http://archive.ubuntu.com trusty-updates InRelease
Err http://archive.ubuntu.com trusty-security InRelease   
Err http://archive.ubuntu.com trusty-proposed InRelease  
Err http://archive.ubuntu.com trusty Release.gpg
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
Err http://archive.ubuntu.com trusty-updates Release.gpg
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
Err http://archive.ubuntu.com trusty-security Release.gpg
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
Err http://archive.ubuntu.com trusty-proposed Release.gpg
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease  
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease  
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease  
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-proposed/InRelease  
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-proposed/Release.gpg  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Container IFCONFIG/PING

➜  code  docker run -it ubuntu /bin/bash
root@7bc182bf87bb:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:04  
          inet addr:172.17.0.4  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:4/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:738 (738.0 B)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@7bc182bf87bb:/# ping google.com
PING google.com (74.125.226.0) 56(84) bytes of data.
64 bytes from lga15s42-in-f0.1e100.net (74.125.226.0): icmp_seq=1 ttl=56 time=12.3 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.367/12.367/12.367/0.000 ms
root@7bc182bf87bb:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=21.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=21.7 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=44 time=21.7 ms

Also, apt-get update fails when I force IPv4:

root@6d925cdf84ad:/# sudo apt-get update -o Acquire::ForceIPv4=true
Err http://archive.ubuntu.com trusty InRelease

Err http://archive.ubuntu.com trusty-updates InRelease

Err http://archive.ubuntu.com trusty-security InRelease

Err http://archive.ubuntu.com trusty-proposed InRelease

Err http://archive.ubuntu.com trusty Release.gpg
  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.153 80]
Err http://archive.ubuntu.com trusty-updates Release.gpg
  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.153 80]
Err http://archive.ubuntu.com trusty-security Release.gpg
  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.153 80]
Err http://archive.ubuntu.com trusty-proposed Release.gpg
  Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.153 80]
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease

For me, it worked after a restart. – ssi-anik Aug 13 '19 at 17:22 — ssi-anik, Aug 13 '19 at 17:22

score 86 · Accepted Answer · edited Aug 25 '17 at 10:32

86

Woo, I found a post on github that solved my problem.

After Steve K. pointed out that it wasn't actually a DNS issue and was a connectivity issue, I was able to find a post on github that described how to fix this problem.

Apparently the docker0 network bridge was hung up. Installing bridge-utils and running the following got my Docker in working order:

apt-get install bridge-utils
pkill docker
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
service docker restart

edited Aug 25 '17 at 10:32

I159

105
6

answered Nov 08 '14 at 19:09

Thomas V.

1,991
2
16
13

1

you don't need to rebulid your images. resolv.conf is generated everytime you run new container. so you need to remove old container and start another one. i was fased this problem yesterday. also, if you are in corporate intranet, you can pass --dns-search=your.company.domain to the docker daemon in /etc/default/docker in DOCKER_OPTS env variable near the --dns --dns flags. – Alexander.Iljushkin Feb 26 '15 at 07:48
3

On arch linux I needed `ip link set down docker0` instead of `ifconfig docker0 down` and `systemctl restart docker` instead of `service docker start`. To delete all images, I did `docker rmi $(docker images -q)` – meshy Sep 09 '15 at 13:59
That worked the first time for me. Then I rebooted, and the problem reappeared: reproducing those steps didn't fix the issue again. I have no idea what this is about. – user626921 Dec 03 '15 at 11:52
1

Just saw that my docker0 interface was down, i executed `/etc/init.d/docker restart` and it's back to business – lolesque Jan 12 '16 at 16:28
1

Issue fixed with these steps, although I had to reinstall docker-engine because docker0 ended up not being found after I followed the steps : pkill docker didn't really stop the service, it may be why it became messy. – JJP May 11 '16 at 11:46
2

Just restart Docker daemon work for me – Nolwennig Jun 07 '18 at 15:32
Like @Nolwennig , I also restarted docker daemon (sudo systemctl restart docker.service) and the problem was gone performing all the steps described in this answer. Might worth giving it a try. – ThomasMX Apr 26 '22 at 15:23

score 28 · Answer 2 · answered Jun 27 '18 at 23:06

If it is a DNS resolver problem, here is the solution:

First thing to check is run cat /etc/resolv.conf in the docker container. If it has an invalid DNS server, such as nameserver 127.0.x.x, then the container will not be able to resolve the domain names into ip addresses, so ping google.com will fail.

Second thing to check is run cat /etc/resolv.conf on the host machine. Docker basically copies the host's /etc/resolv.conf to the container everytime a container is started. So if the host's /etc/resolv.conf is wrong, then so will the docker container.

If you have found that the host's /etc/resolv.conf is wrong, then you have 2 options:

Hardcode the DNS server in daemon.json. This is easy, but not ideal if you expect the DNS server to change.
Fix the hosts's /etc/resolv.conf. This is a little trickier, but it is generated dynamically, and you are not hardcoding the DNS server.

1. Hardcode DNS server in docker daemon.json

Edit /etc/docker/daemon.json
```
{
    "dns": ["10.1.2.3", "8.8.8.8"]
}
```
Restart the docker daemon for those changes to take effect:
sudo systemctl restart docker
Now when you run/start a container, docker will populate /etc/resolv.conf with the values from daemon.json.

2. Fix the hosts's /etc/resolv.conf

A. Ubuntu 16.04 and earlier

For Ubuntu 16.04 and earlier, /etc/resolv.conf was dynamically generated by NetworkManager.
Comment out the line dns=dnsmasq (with a #) in /etc/NetworkManager/NetworkManager.conf
Restart the NetworkManager to regenerate /etc/resolv.conf :
sudo systemctl restart network-manager
Verify on the host: cat /etc/resolv.conf

B. Ubuntu 18.04 and later

Ubuntu 18.04 changed to use systemd-resolved to generate /etc/resolv.conf. Now by default it uses a local DNS cache 127.0.0.53. That will not work inside a container, so Docker will default to Google's 8.8.8.8 DNS server, which may break for people behind a firewall.
/etc/resolv.conf is actually a symlink (ls -l /etc/resolv.conf) which points to /run/systemd/resolve/stub-resolv.conf (127.0.0.53) by default in Ubuntu 18.04.
Just change the symlink to point to /run/systemd/resolve/resolv.conf, which lists the real DNS servers:
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
Verify on the host: cat /etc/resolv.conf

Now you should have a valid /etc/resolv.conf on the host for docker to copy into the containers.

Thanks for this, was really going loosing my mind trying to understand what was happening with docker containers and 18.04 resolving IPs on a VPN. Fixing /etc/resolv.conf for 18.04 worked for me! — George Papas, Jul 23 '18 at 04:41
Surely 2B isn't going to survive an update of the `systemd` package... — Auspex, May 23 '19 at 14:50
I had to reboot my VM after changing the symlink. Restarting the network might have been sufficient (I didn't try that). — Daniel, Mar 13 '20 at 18:51

score 13 · Answer 3 · edited Feb 23 '18 at 20:44

In an attempt to add additional value to an issue I also experienced; with an alternative answer:

My network was office related and Google DNS settings were blocked so that the container could ping IP addresses but not domain names.

My host's /etc/resolv.conf originally looked like;

#Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search companyDomain.co.za

This is due to Network Manager doing some kind of masking of the DNS server details.

Unfortunately according to the docker manuals docker will filter out any localhost IP addresses when building the container's resolv.conf and replace them with Google's DNS IPs. Which in my case caused domain names to be off-limits.

I had to:

Reset my /etc/default/docker to default so containers use my host's resolv.conf content instead.
Edit /etc/NetworkManager/NetworManager.conf and comment out the line dns=dnsmasq. This is so NM can specify the actual DNS IP addresses instead of 127.0.0.1.
Restart NM with sudo service network-manager restart.
Restart docker service with sudo service docker restart.

Running a container would then allow it to do apt-get update/upgrade, for example.

This actually worked for me. And I was behind company intranet — Daniel Andrei Mincă, Sep 29 '16 at 10:24
This solution works great for Ubuntu 16.04 and earlier. For Ubuntu 18.04 and later, see https://serverfault.com/a/918568 — wisbucky, Jun 27 '18 at 23:07

score 9 · Answer 4 · edited Feb 18 '19 at 15:44

9

Docker official doc gives instruments to configure a DNS server for use by Docker

Open the /etc/default/docker file for editing:
```
sudo nano /etc/default/docker
```
Add a setting for Docker:
```
DOCKER_OPTS="--dns 8.8.8.8"
```
Replace 8.8.8.8 with a local DNS server such as 192.168.1.1. You can also specify multiple DNS servers. Separated them with spaces, for example:
```
--dns 8.8.8.8 --dns 192.168.1.1
```
Warning: If you're doing this on a laptop which connects to various networks, make sure to choose a public DNS server.

PS: nm-tool can be used to check local host DNS server
Save and close the file.
Restart the Docker daemon.
```
sudo service docker restart
```

edited Feb 18 '19 at 15:44

Slava Fomin II

1,661
4
17
22

answered May 27 '15 at 05:48

tryer3000

191
1
2

Note that this the old config file for Docker Upstart and SysVinit. The current way for systemd (since Ubuntu 16.04) is to use [`/etc/docker/daemon.json` for docker daemon settings](https://docs.docker.com/engine/reference/commandline/dockerd//#daemon-configuration-file) such as dns. – wisbucky Jun 27 '18 at 23:19

score 8 · Answer 5 · answered Nov 08 '14 at 18:38

8

Your error is here:

 Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19).
 connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]

This isn't an error with DNS, instead your system is trying to connect to IPv6 hosts and failing . Presumably because you don't have IPv6 access on your host. The actual lookup of the IPv6 address succeeds. (The ubuntu mirror/archive is available over both IPv6 and IPv4. You were just unlucky enough to hit an IPv6 one because your system believes it should work.)

You should either fix that, by installing miredo, or retry until your hit an IPv4 mirror.

Again the important thing to realize here is that DNS is not to blame, as you can see by your own ping tests.

answered Nov 08 '14 at 18:38

1

Thanks for the fast reply and clarifying that it's not actually a DNS issue, I appreciate it. I installed miredo -- no go. It's also worth noting that when I run apt-get update -o Acquire::ForceIPv4=true apt-get update still fails, I've updated my original post with that reply. I've tried disabling UFW thinking maybe that was the case, and still haven't had luck. – Thomas V. Nov 08 '14 at 18:47
Weird - you can see you have IPv4 connectivity because your ping succeeds. But you can't connect to the mirror regardless suggests you have some odd routing/networking issue (which I guess is why you're posting here!) – Nov 08 '14 at 19:01

score 1 · Answer 6 · answered Jul 11 '20 at 02:42

I encountered this problem when I allowed the Ubuntu installer to install the Docker snap package. When I ditched that and switched to the official Docker package the problem resolved itself.

sudo snap remove docker
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

score 1 · Answer 7 · answered Jun 23 '15 at 01:42

For other readers who come here while using boot2docker, here is how I fixed. In fact, the answer above pointed me to the right direction.

Basically, for some reason containers inside boot2docker couldn't resolve hostnames.

So I just restarted boot2docker and started the containers. Now hostnames can resolve properly again.

I suppose the problem was starting boot2docker while network on the host was being connected which caused boot2docker to start up and enter into a non-working state.

score 0 · Answer 8 · edited Jul 08 '20 at 08:19

0

Open the file /lib/systemd/system/docker.service

In ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock add
--dns 8.8.8.8

Like this:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --dns 8.8.8.8

Restart docker service

edited Jul 08 '20 at 08:19

Stuggi

3,366
4
17
34

answered Jul 06 '20 at 12:14

Anish Varghese

101
1

score 0 · Answer 9 · answered Dec 17 '20 at 09:16

0

I had the same issue. I imagine it's to do with docker bridge network as other people mentioned. Re-installing docker worked for me on Fedora:

sudo dnf remove docker-ce
sudo dnf install docker-ce

Re-installing probably sets up bridge network again

answered Dec 17 '20 at 09:16

oradwell

101
2

score 0 · Answer 10 · answered Nov 10 '21 at 08:45

0

Use --network host in the command-line. Or use a docker-compose file with this option.

answered Nov 10 '21 at 08:45

X99

172
2
14

score 0 · Answer 11 · answered Aug 11 '16 at 22:16

0

I had the same issue on Windows. This command got it working for me: docker-machine restart

answered Aug 11 '16 at 22:16

speedplane

111
4

michi.0x5d · Answer 12 · 2017-08-10T15:53:28.370

0

Had a similar issue, but also name resolving between containers inside a User defined network seemed to be a bit flaky. Some couldn't resolve anything like you.

The issue was a moved /var/lib/docker. For space reasons it was mounted via nfs. Adding a local filesystem and moving the files there resolves the issue.

edited Aug 10 '17 at 15:53

answered Aug 10 '17 at 10:54

michi.0x5d

154
8

If you think a question can be answered by an answer on a similar question, please mark it as a duplicate of that question. If you can't do that, you should leave a comment rather than making it a separate answer. – Jenny D Aug 10 '17 at 11:32

score 0 · Answer 13 · answered Jun 07 '18 at 15:30

0

Restart the Docker daemon on Debian9

service docker restart

and the connections and networks works fine

answered Jun 07 '18 at 15:30

Nolwennig

340
3
8

Docker containers can't resolve DNS on Ubuntu 14.04 Desktop Host

13 Answers13

Linked