0

When there are lots of external VMs which are accessible only via a firewall and there are multiple namespaces in the cluster, each with its own set of external VMs, you end up with a lot of ServiceEntries, which in turn cause a lot of DNS queries for

$host.$ns.svc.cluster.local 
$host.svc.cluster.local
$host.cluster.local

where $host is for example: vm-1.vm-cluster.example.com with let's say 10 namespaces and 100 pods in each namespace, that's quite a number of DNS queries, all answered with NXDOMAIN, so the local istio DNS proxy cache does not work. This can lead to an overload situation of the coreDNS Pods (the memory size, including buffers goes up and cause the DNS Pod to OOM).

My question is: how can I prevent those queries? The solutions I came up with are:

Replacing $host with a FQDN, which includes the trailing dot, does not work because the X.509 certs contain the hostname without the trailing dot. Change the DNS config of each Pod and set the ndots to something like 3. But this sounds like a maintenance nightmare. I would like to solve this via a clever combination of ServiceEntry, VirtualService and Gateway definitions.

K8s version 1.19 (vmware edition) Istio version 1.9 (also vmware edition)

bofh.at
  • 101
  • Is OOM only the problem or is something else going on? Can you e.g. increase memory resources for DNS Pod? – Mikołaj Głodziak Oct 27 '21 at 06:58
  • I can't reliably increase the memory, because the VMware cluster has a fixed config for coreDNS, i was told. It's also a latency problem. the bigger the queues, the longer a DNS query takes. you want them to be fast. I found a problem with our config though. Somehow all services got defined in all namespaces. We can and will define the services only in those namespaces where they are needed. But i can see that the problem will come back later, if the cluster and it's external dependencies grow. – bofh.at Oct 27 '21 at 22:29
  • Could you add your yaml files (current configuration) to reproduce your problem? – Mikołaj Głodziak Oct 29 '21 at 12:13
  • Hello @bofh.at. Any updates? – Wytrzymały Wiktor Nov 03 '21 at 13:29

0 Answers0