2

My current setup involves an EKS Cluster with multiple namespaces (multi-tenant) across many different EKS nodes in private subnets. I would like the egress traffic from the pods to have a dedicated EIP per namespace. AFAIK there are no off the shelf solutions available for this problem. I have searched long and hard on the internet but in vain. Here are some of the solutions that I have tired, but eventually hit a roadblock.

Istio Egress Gateway Allows you to direct all outbound traffic from pods via a dedicated egress gateway pod. But the egress gateway pod will assume the ip of the node on which it is running, which will not work for my use case. Also, I have not found decent documentation around how I can setup multiple egress gateway pods across multiple nodes.

Calico Egress Gateway Very similar to Istio Egress Gateway solution and the same constraint applies

Custom Solution I have started implementing a custom solution, where I setup multiple Gateway Nodes in public subnet each with a dedicated EIP. I can now modify the IP routes/gateways in the private EKS node, to route traffic via a specific gateway node based on the pod source IP. This solution feels very kludgy and the operational overhead of such a solution is very high.

I have looked at solutions like this, but have not had any luck with them.

Is there a better approach/solution to this problem?

P.S. My production cluster is extremely large, I cannot afford to stand up a cluster for each namespace.

I have cross posted this question on stackoverflow as well. I am really not sure which forum is a better place this question. Happy to delete it where appropriate.

Dave M
  • 4,494
  • 21
  • 30
  • 30
Karthik Balasubramanian
  • 121
  • 3

2 Answers2

2

Summary from the discussion on this topic for more visibility here:

There is a feature request created in GitHub, but no updates since March.

Calico Enterprise would solve for this specific use case by assigning an IP to the namespace as described in their integration for Kubernetes here:

Egress Gateway defines a static egress IP for SNAT on traffic leaving the cluster and applies it to a specific namespace. Then it designates one pod in a namespace as the egress pod and assigns a routable IP which is used specifically for egress traffic leaving the cluster.

But in practice, Egress gateway pod is not AWS VPS aware, hence it cannot assign EIPs to Egress pods. When the traffic exits out of gateway pods it assumes the IP of the Node itself as the source IP.

many different EKS nodes in private subnets...

EIP does not work in private subnet. If you have worker nodes designated in public subnet, you can use EIP and reverse proxy on these public accessible nodes to expose services in private network. You can then use Calico Egress Gateway on these public nodes for the reverse proxy pods in turn use the underlying EIP.

Bazhikov
  • 138
  • 3
0

I finally settled on solution, where i egress traffic from pods within private subnet via "Proxy Pods" that I setup in public subnets. These proxy pods are assigned public ips, hence the egress traffic automatically assume these ips. Operationally there is some overhead in this solution, but this is the best i could come up with given the constraints

Karthik Balasubramanian
  • 121
  • 3