1

I would like to have a Service of LoadBalancer type that points to an nginx on port 80, and point to a separate sshd pod for port 22. However I can't route based on the port with a single selector.

The use case I'm going for is similar to github.com which accepts traffic to github.com for port 80 as well as 22 for ssh traffic. So DNS would point to a set of k8s loadbalancers in this case that I would assume would route to the appropriate pods per port. So if I'm going at it the wrong way, lemme know. I'm open to other solutions.

What I would like to avoid doing is setting up a separate pod like HAProxy that routes per port.

I looked into using an Ingress, but that is only for HTTP traffic.

An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

https://kubernetes.io/docs/concepts/services-networking/ingress/

Matt
  • 121
  • 5
  • Hi Matt, could you please correct your phrase 'However I can't route based on the port w/ a single selector' ? What is 'w/'? – mozello Feb 21 '22 at 11:30
  • 1
    Are you aware that Kubernetes Ingress can expose TCP ports as described [here](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/)? How did you set up your k8s cluster (cloud provider or on-premise)? What k8s version are you using? – mozello Feb 21 '22 at 20:10
  • @IvanM. I am currently running this locally using kind which is on kubernetes v1.23.3. To get the LoadBalancer to work, i'm using metallb.Let me check out that link, though. I see that the nginx controller extends the default ingress controller of kubernetes. – Matt Feb 21 '22 at 21:51

2 Answers2

1

What I would like to avoid doing is setting up a separate pod like HAProxy that routes per port.

You don't need to set up such separate Pod.

The Kubernetes Ingress by default does not support TCP or UDP services. But for example, ingress-nginx controller provides a mechanism to support TCP or UDP on different ports. You can expose TCP or UDP ports by modifying ConfigMaps.

For this reason, this Ingress controller uses the flags '--tcp-services-configmap' and '--udp-services-configmap' to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format:

<namespace/service name>:<service port>:[PROXY]:[PROXY]

Check additional info here.

Such ConfigMap should already be available before deploying the Ingress Controller.

So, try to:

1. Create a ConfigMap with the following TCP service configuration.

$ cat ingress-nginx-tcp.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-tcp
  namespace: default  
data:
  "22": targetnamespace/target-service:22

2. Point Ingress controller to this ConfigMap using the --tcp-services-configmap flag in the configuration like this:

$ kubectl get deployment ingress-nginx-controller -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: default
spec:
...
  template:
...
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp
...

3. Expose port 22 in the Service defined for the Ingress like this:

$ kubectl get svc ingress-nginx-controller -o yaml 
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: default
spec:
  ports:
  - name: tcp-22
    nodePort: 30957
    port: 22
    protocol: TCP
    targetPort: 22
...
  type: LoadBalancer
...

You can define any number of ports that can be exposed using this method.

There is another option for those who are using an ingress-nginx helm chart. Most of the configuration is already done, and you just need to specify your ports in tcp section like this:
tcp: 
  2222: "default/example-tcp-svc:22"

where 2222 is the exposed port and 22 is the service port.

mozello
  • 121
  • 4
0

Thanks to @Ivan M. who pointed me in the right direction, I just want to add a code example of the solution here:

First of all, the confusing thing is that the kubernetes documentation lead me down the wrong path thinking that you can't use an ingress for non http traffic. The thing that you have to know about is that you can use a different ingress controller. With the nginx ingress controller, you can proxy arbitrary tcp or even udp traffic.

The second hurdle is that there are two separate repos for the nginx ingress. Those are:

I ended up using the kubernetes/ingress-nginx controller.

nginx-values.yaml

controller:
  service:
    enableHttp: true

tcp:
  "22": "other-namespace/service-name:22"

Then we have the helm install

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm -n nginx -f nginx-values.yaml install ingress-nginx ingress-nginx/ingress-nginx

Then I install the Ingress into k8s. Notice the ingressClassName

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: nginx
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: my-http-service
      port:
        number: 80

And then you kubectl that bad boy

kubectl apply -f ingress.yaml
Matt
  • 121
  • 5