2

We are just getting started with k8s. One of the things we need to do is expose a service running on each bare metal Ubuntu 20.04 host node to pods in k8s. The service is listening on the node's real IP as well as localhost on port 8500. Is there an easy way to do this?

I can get to port 80 and 443 on the node from inside a pod using its IP but not to 8500.

David Tinker
  • 557
  • 1
  • 8
  • 16
  • 3
    Are you sure that the service is listening on `0.0.0.0:8500`? – kupson May 24 '21 at 19:15
  • Yes I tested from another server: curl -v http://:8500 works. Same thing from inside pod just sits on "Trying..." – David Tinker May 25 '21 at 05:45
  • I can get to the same service running on other nodes from inside the pod, just not to the instance running on its own node. So maybe this is some sort of security thing to not allow the pod to access random ports on its node? – David Tinker May 25 '21 at 06:01
  • Please check if your port 8500 is open. [Here](https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/) you can find information how to do this in several ways. Additionally, did you get any error when did you try to access node on port 8500? – p10l May 25 '21 at 12:48
  • Yes I have checked. I can get to the service from the host with curl to its public ip. I can get to the service from other hosts with its public ip. I cannot get to it from k8s pods running on the host (node) using its public ip. I can get to it from k8s pods running on other hosts using its public ip. – David Tinker May 26 '21 at 08:37
  • Same thing happens if I deploy service in a pod using hostNetwork:true. Other pods on the same node cannot get it it, no problem from other nodes. – David Tinker May 26 '21 at 08:38
  • 1
    I have finally made some progress on this: "iptables -A INPUT -p tcp --dport 8500 -j ACCEPT" sorts it out. I didn't think that connections from inside k8s pods on the same host would get blocked by our firewall rules. Now I just need to figure out how to poke a more specific hole in the firewall. – David Tinker May 26 '21 at 09:06
  • "iptables -A INPUT -i cni0 -j ACCEPT" seems to work – David Tinker May 26 '21 at 10:00

1 Answers1

3

As @David Tinker mentioned in the comments, problem is solved.

connections from inside k8s pods on the same host would get blocked by our firewall rules.

So, the solution is hidden in iptables:

iptables -A INPUT -p tcp --dport 8500 -j ACCEPT
iptables -A INPUT -i cni0 -j ACCEPT
Mikołaj Głodziak
  • 323
  • 1
  • 10
  • 2
    This was helpful to understand k8s networking: https://itnext.io/kubernetes-journey-up-and-running-out-of-the-cloud-flannel-c01283308f0e – David Tinker May 27 '21 at 08:29