New to GCloud and GKE and having a frustrating time with DNS.
We have a VPN between our office and GCloud running a Shared VPC. Existing firewall rules seem to work fine. We can ping both ways, we can ssh to Google successfully.
So now from within GKE, we need to be able to resolve hostnames across the VPN using DNS. Should be simple.
I edited the kube-dns config map and added our internal domain name using stubDomains pointing to our two DNS servers. After the kube-dns pods get redeployed, I verified that in the logs, they are getting the new stubDomain section. However I still can't resolve any hosts, even from the kube-dns containers themselves.
While logged into dnsmasq container:
/etc/k8s/dns/dnsmasq-nanny # cat stubDomains
{"internal.domain.com": ["10.85.128.5", "10.85.128.6"]}
/ # nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve
Name: google.com
Address 1: 108.177.9.138 ox-in-f138.1e100.net
Address 2: 108.177.9.101 ox-in-f101.1e100.net
Address 3: 108.177.9.139 ox-in-f139.1e100.net
Address 4: 108.177.9.100 ox-in-f100.1e100.net
Address 5: 108.177.9.102 ox-in-f102.1e100.net
Address 6: 108.177.9.113 ox-in-f113.1e100.net
Address 7: 2607:f8b0:4003:c13::71 ox-in-x71.1e100.net
/etc/k8s/dns/dnsmasq-nanny # cd /
/ # nslookup rancher.internal.domain.com
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'rancher.internal.domain.com': Name does not resolve
nslookup: can't resolve 'rancher.internal.domain.com': Name does not resolve
/ # nslookup rancher.internal.domain.com 10.85.128.5
Server: 10.85.128.5
Address 1: 10.85.128.5
nslookup: can't resolve 'rancher.internal.domain.com': Name does not resolve
I can see in the VPC firewall logs that queries are being "allowed" through the VPC firewall, but never arriving at our VPN.
I can ping the DNS server from within the dnsmasq container, and I can even curl web pages on the VPN side of the network without issues.
I can bring up a Google Compute node and can access the VPN DNS server without a problem.
It's only DNS that won't work and only from GKE and its containers.
Any thoughts?
