VPN Conflicts With Docker

01 April 2019

Written by

Mark Wallsgrove
Senior Engineer

It turns out that Docker has an internal Domain Name Service (DNS). Did you know? It’s new to me too! I learnt the hard way while using a VPN.

This is the error that I found within a container:

persona_1_8a84f3f41190 | 2019/02/25 08:51:14.138610 [WARN] (view) kv.list(global):
Get http://consul:8500/v1/kv/global?recurse=&stale=&wait=60000ms: dial tcp: lookup
consul on 127.0.0.11:53: no such host (retry attempt 12 after "1m0s")

The error states that the domain name consul, which is the name of one of my containers, couldn’t be found by using the DNS at 127.0.0.11 on port 53. But I don’t run a DNS at 127.0.0.11?

Searching for 127.0.0.11:53 led me to Docker - Configure DNS which states: Note: The DNS server is always at 127.0.0.11 Huh, OK, I guess Docker has an internal DNS. And sure enough, I can connect to the service from within a container.

The error seems to go away if I stop the VPN and recycle the containers, how odd.

The reason I was using the VPN was to be able to SSH into an AWS EC2 instance, which is only accessible through the VPN. I wonder if the VPN alters my host’s DNS settings?

cat /etc/resove.conf
nameserver 172.16.0.23
nameserver 127.0.0.53

I have two DNS entries, one for my local Stubby DNS 127.0.0.53, and 172.16.0.23. Who is 172.16.0.23?

A quick search shows that AWS uses 172.16.0.23 as their internal DNS. The reason why we would want to use their DNS would be to resolve internal domain names. 172.16. is an internal IP address which is only accessible through the VPN.

This leaves two questions, which public DNS does Docker use to resolve DNS queries and why would my VPN configuration affect it?

By default, a container inherits the DNS settings of the Docker daemon, including the /etc/hosts and /etc/resolv.conf. You can override these settings on a per-container basis.

This would mean that any DNS queries to 127.0.0.11 would use the private AWS DNS 172.16.0.23 that my VPN desires, which would result in a timeout as Docker isn’t using my VPN.

The solution Fix Docker Networking DNS suggests overriding the DNS to use:

/etc/docker/daemon.json

{
    "dns": ["8.8.8.8", "8.8.4.4"]
}

By setting the DNS IP addresses to a public DNS address we avoid the issue of inheriting a DNS address which is not accessible due to the traffic not being routed through the VPN.

The configuration above resolves the timeout issue. It would seem as if the internal Docker DNS will stop attempting to resolve the request (even the internal DNS names such as the container name) if any of the supplied DNS entries fail with a timeout.

We are still early in the journey of applying docker to our micro service archiecture. In the future we will follow up with more articles covering different subjects regarding Docker & our next adventure, Kubernetes.