Exposing simple pod using kubernetes ingress

Question

Hi I'm learning kubernetes and I'm having trouble exposing the service. I want to route traffic to my cluster from HAProxy. I'm using my own bare-metal server.

EDIT: I've also created an ingress controller.

Now, when I describe my ingress I can see IP Address of worker machine but still I've got Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) And no idea how to get access to my pod...

example config:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache
  labels:
    app: apache-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apache-test
  template:
    metadata:
      labels:
        app: apache-test
    spec:
      containers:
      - name: apache
        image: httpd
        ports:
        - containerPort: 80

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: apache-test-service
spec:
  selector:
    app: apache-test
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    name: http

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apache-test-ingress
spec:
  rules:
  - host: apache-test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: apache-test-service
            port:
              number: 80

What's wrong?

describe ingress:

Name:             apache-test-ingress
Namespace:        default
Address:          192.168.6.72
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host         Path  Backends
  ----         ----  --------
  apache-test  
               /   apache-test-service:80 (10.44.0.1:80)
Annotations:   <none>
Events:        <none>

describe service:

Name:              apache-test-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=apache-test
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.104.63.167
IPs:               10.104.63.167
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.44.0.1:80
Session Affinity:  None
Events:            <none>

describe controller:

Name:         ingress-nginx-controller-55bc4f5576-vpsgb                                                                                                                                                    
Namespace:    ingress-nginx
Priority:     0
Node:         kubernetes-node02/192.168.6.72
Start Time:   Sun, 16 May 2021 16:47:26 +0200
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx 
              pod-template-hash=55bc4f5576
Annotations:  <none>
Status:       Running
IP:           10.36.0.1
IPs:
  IP:           10.36.0.1
Controlled By:  ReplicaSet/ingress-nginx-controller-55bc4f5576
Containers:
  controller:
    Container ID:  docker://7daf566a039aba0d06f856b0adcc03659423ec2462c33d9a79f820b58dfcbf98
    Image:         k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
    Image ID:      docker-pullable://k8s.gcr.io/ingress-nginx/controller@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 16 May 2021 16:47:28 +0200
    Ready:          True
Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-55bc4f5576-vpsgb (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftnfs (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-ftnfs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

logs from POD doesn't show anything... logs from ingress controller:

I0516 14:47:28.871207       8 flags.go:208] "Watching for Ingress" class="nginx"
W0516 14:47:28.871287       8 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0516 14:47:28.872068       8 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0516 14:47:28.872594       8 main.go:241] "Creating API client" host="https://10.96.0.1:443"
I0516 14:47:28.887394       8 main.go:285] "Running in Kubernetes cluster" major="1" minor="21" git="v1.21.0" state="clean" commit="cb303e613a121a29364f75cc67d3d580833a7479" platform="linux/amd64"
I0516 14:47:29.768986       8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0516 14:47:29.772688       8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0516 14:47:29.775841       8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0516 14:47:29.793896       8 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0516 14:47:29.829161       8 nginx.go:254] "Starting NGINX Ingress controller"
I0516 14:47:29.848934       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"0cf6bc98-71b3-4387-a535-7d3dcb956fc8", APIVersion:"v1", ResourceVersion:"401441", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0516 14:47:30.936661       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"apache-test-ingress", UID:"6e3c5757-28cf-4a68-be98-827fd69ee86f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"400092", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0516 14:47:31.030103       8 nginx.go:296] "Starting NGINX process"
I0516 14:47:31.030266       8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0516 14:47:31.030658       8 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0516 14:47:31.031274       8 controller.go:146] "Configuration changes detected, backend reload required"
I0516 14:47:31.040799       8 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0516 14:47:31.041189       8 status.go:84] "New leader elected" identity="ingress-nginx-controller-55bc4f5576-vpsgb"
I0516 14:47:31.054203       8 status.go:204] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-55bc4f5576-vpsgb" node="kubernetes-node02"
I0516 14:47:31.129614       8 controller.go:163] "Backend successfully reloaded"
I0516 14:47:31.129922       8 controller.go:174] "Initial sync, sleeping for 1 second"
I0516 14:47:31.130053       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-55bc4f5576-vpsgb", UID:"16d9fca9-8ac9-4fc1-be40-056540857035", APIVersion:"v1", ResourceVersion:"401513", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0516 14:48:31.054140       8 status.go:284] "updating Ingress status" namespace="default" ingress="apache-test-ingress" currentValue=[] newValue=[{IP:192.168.6.72 Hostname: Ports:[]}]
I0516 14:48:31.067947       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"apache-test-ingress", UID:"6e3c5757-28cf-4a68-be98-827fd69ee86f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"401625", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

describe POD

Name:         apache-67487b7c8b-8jbgb
Namespace:    default
Priority:     0
Node:         kubernetes-node01/192.168.6.71
Start Time:   Sun, 16 May 2021 15:13:07 +0200
Labels:       app=apache-test
              pod-template-hash=67487b7c8b
Annotations:  <none>
Status:       Running
IP:           10.44.0.1
IPs:
  IP:           10.44.0.1
Controlled By:  ReplicaSet/apache-67487b7c8b
Containers:
  apache:
    Container ID:   docker://70e4e3c4e01dffa11aa3c945f297e2cf3bc8af249c8d900c8aa30381ce7f56e6
    Image:          httpd
    Image ID:       docker-pullable://httpd@sha256:e4c2b93c04762468a6cce6d507d94def02ef4dc285278d0d926e09827f4857db
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 16 May 2021 15:13:10 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c8dfx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-c8dfx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

EDIT: I used an ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/baremetal/deploy.yaml

Hi Paweł Zając welcome to S.F. Be aware that traffic is only routed to `Ready` pods, so if you apache is in CLBO or similar, it will also not work. You'll want some combination of `kubectl describe` and `kubectl logs` across the components to see what the misalignment is. No one can help you with the minimum level of detail you've provided here. Good luck! — mdaniel, May 16 '21 at 18:07
Hi, thanks for your answer. I've added more details. I want to be sure: curl 192.168.6.72 -H 'Host apache-test' should give me right output? — Paweł Zając, May 16 '21 at 20:42
Not as written, because your `-H` is missing the colon separator between the `Host:` header and its value -- but otherwise, yes. If that for some reason does not pan out, you may next try `curl -vH 'host: apache-test' http://10.36.0.1` to go directly to the Ingress controller's Service. All of that, of course, should happen only after, as I originally said, checking that the Pod is `Ready` and otherwise behaving sanely: `curl -v 10.44.0.1` and `curl -v 10.104.63.167` — mdaniel, May 16 '21 at 22:32
Pod says it's running (I've added full describe) When I'm trying all of this curl commands...still no output. Trying to use curl from pod with apache, it returns 200, so it's working...but only inside cluster... — Paweł Zając, May 16 '21 at 23:16
I find the assertion "No output" from a curl with `-v` highly suspicious. Since you don't want to include the output of your own investigation, I think this "troubleshooting via comments" is now over. Good luck with your question — mdaniel, May 16 '21 at 23:18
I meant this effect: ```~ $ curl -vH 'host: apache-test' http://10.36.0.1 * Trying 10.36.0.1:80... * TCP_NODELAY set * connect to 10.36.0.1 port 80 failed: Connection timed out * Failed to connect to 10.36.0.1 port 80: Connection timed out * Closing connection 0 curl: (28) Failed to connect to 10.36.0.1 port 80: Connection timed out ``` and when I'm using address of cluster: ``` ~ $ curl http://192.168.6.72/ -H 'Host:apache-test' curl: (7) Failed to connect to 192.168.6.72 port 80: Connection refused ``` — Paweł Zając, May 16 '21 at 23:23
Hi, do I understand correctly that you have a `HAProxy` before your Kubernetes cluster and you have installed `Nginx Ingress` controller inside of your cluster? Could you please tell how exactly you've installed your `Ingress` controller? I can't see how you are exposing your `Ingress` controller to the external access so please add the `Service` definition associated with it. — Dawid Kruk, May 17 '21 at 11:09
Yes, HAProxxy is just proxy server for my network. I used Ingress Controller from official docs: https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal — Paweł Zając, May 17 '21 at 16:00

score 2 · Accepted Answer · answered May 18 '21 at 13:54

TL;DR

Instead of using the port of 80/443, you should use the ports that are associated with the Service of type NodePort that you've created during the Ingress controller provisioning.

$ kubectl get services -n

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.233.39.209   <none>        80:30983/TCP,443:32636/TCP   4h30m
ingress-nginx-controller-admission   ClusterIP   10.233.54.211   <none>        443/TCP                      4h30m

In this example, you should use the IP address of one of your Nodes and the corresponding port (when trying to connect to it from outside):

curl http://IP_ADDRESS:30983 or curl -v -k https://IP_ADDRESS:32636

Explanation

Focusing on the part of the YAML manifest that you've used:

https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/baremetal/deploy.yaml

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    <-- REDACTED --> 
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort # <-- IMPORTANT 
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

When you applied the whole manifest you've created a Service of type NodePort.

Citing the official documentation:

NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.

-- Kubernetes.io: Docs: Concepts: Services networking: Service: Publishing services service types

Type Nodeport

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767). Each node proxies that port (the same port number on every Node) into your Service. Your Service reports the allocated port in its .spec.ports[*].nodePort field.

-- Kubernetes.io: Docs: Concepts: Services networking: Service: Nodeport

This NodePort is the entrypoint for your Ingress controller. You will need to send a request to it's ports to contact your Ingress controller (and then Ingress controller will route the traffic accordingly to the Ingress resource).

You can check to which port you should send the traffic to by invoking (previously mentioned):

$ kubectl get services -n

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.233.39.209   <none>        80:30983/TCP,443:32636/TCP   4h30m
ingress-nginx-controller-admission   ClusterIP   10.233.54.211   <none>        443/TCP                      4h30m

In this example it should be following:

curl http://IP_ADDRESS:30983 for HTTP
curl -v -k https://IP_ADDRESS:32636 for HTTPS

If you would like to expose your Nginx Ingress controller on a port 80/443 you could opt to use Service of type LoadBalancer that is backed with:

Metallb.universe.tf

Metallb will allow you to create a pool of IP address that will be available for allocation for Service of type LoadBalancer.

A side note!

Please remember that with the Ingress resource you've specified, you should send a request with a Host: apache-test.com otherwise you will get a 404. For testing purposes you can set:

- host: instead of:

- host: apache-test.com

Addressing following part of the question:

Now, when I describe my ingress I can see IP Address of worker machine but still I've got Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) And no idea how to get access to my pod...

This won't limit your ability to connect to your Services as this is a resource to send the traffic to when it doesn't match any of the rules in the Ingress resource.

Additional resources:

Hello @PawełZając and welcome to ServerFault! Please remember to [react to answers for your questions](https://stackoverflow.com/help/someone-answers). That way we know if the answers were helpful and other community members could also benefit from them. Try to [accept answer](https://stackoverflow.com/help/accepted-answer) that is the final solution for your issue, upvote answers that are helpful and comment on those which could be improved or require additional attention. Enjoy your stay! — Wytrzymały Wiktor, May 19 '21 at 11:57
Thanks a lot for the clarification. I implement and learn further :) — Paweł Zając, May 20 '21 at 17:14

Exposing simple pod using kubernetes ingress

1 Answers1

Explanation

Type Nodeport