My question is about Kubernetes' internal traffic routing and probably about routing / NAT in general.
Goal:
I'm running a Wireguard service on Kubernetes and I'd like to route all outgoing traffic which has entered through Wireguard through another service (let's call it filter) before it hits the internet. Furthermore I want to be able to transparently exchange the the filter Pods without any disruptions on the client side.
Networkwise I'd like to route the traffic through the following hops:
WG Client -> Wireguard SVC (10.43.112.165) -> Wireguard Pod (10.42.0.32) -> Filter SVC (10.43.111.132) -> Filter Pod (10.42.0.44) -> Internet
What I accomplished to so far is:
WG Client -> Wireguard SVC (10.43.112.165) -> Wireguard Pod (10.42.0.32) -> Filter Pod (10.42.0.44) -> Internet
In order to get there I manually did the following
inside the Wireguard pod
ip route replace default via 10.42.0.44 # the filter pod's IP
inside the filter pod
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
As mentioned above that is not exactly what I want, since I'm not able to exchange the filter pod without reconfiguring the Wireguard Pod.
The problem is that I cannot simply define the filter service as the default gateway in the Wireguard pod.
ip route replace default via 10.43.111.132 # the filter SVC's IP
Error: Nexthop has invalid gateway.
The above makes perfect sense to me since the filter SVC is located in a different network. Yet I don't know how to get around the problem.
In conclusion two questions arise:
- How can I route traffic through another service before it's send out to the internet?
- What would be the proper way to implement this? Can I convince Kubernetes to apply the routing for me when the Wireguard Pod starts up? Or should I maybe bypass Kubernetes completely and simply apply the rules myself in the container's entrypoint or something?