1

Service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    io.kompose.service: grafana
  name: grafana
spec:
  ports:
    - name: "3000"
      port: 3000
      targetPort: 3000
  selector:
    io.kompose.service: grafana
status:
  loadBalancer: {}

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.22.0 (HEAD)
  creationTimestamp: null
  labels:
    io.kompose.service: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: grafana
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert
        kompose.version: 1.22.0 (HEAD)
      creationTimestamp: null
      labels:
        io.kompose.service: grafana
    spec:
      containers:
        - env:
            - name: GF_SERVER_DOMAIN
              #              value: "testing.esamir.com"
              value: "direct.esamir.com"
            - name: GF_SERVER_ROOT_URL
              value: "%(protocol)s://%(domain)s:%(http_port)s/"
            - name: GF_SERVER_SERVE_FROM_SUB_PATH
              value: "true"
            - name: GF_DATABASE_URL
              valueFrom:
                configMapKeyRef:
                  name: grafanaconfig
                  key: url

          image: grafana/grafana:7.4.0-ubuntu
          name: grafana
          ports:
            - containerPort: 3000
          resources: {}
      restartPolicy: Always
status: {}

ingress-nginx config:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-kubernetes-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: testing1.com
      http:
        paths:
          - backend:
              serviceName: hello-kubernetes-second
              servicePort: 80
    - host: testing2.com
      http:
        paths:
          - backend:
              serviceName: dashboard
              servicePort: 3000

if I connect to http or https on testing1.com which is running a simply hello-world kubernetes app. Everything works flawlessly.

If I connect to testin2.com on https or http, I get the 503 error. "503 Service Temporarily Unavailable"

Looking at the logs, I found this error: "2021-02-09 15:34:25.309 PSTError obtaining Endpoints for Service "test/dashboard": no object matching key "test/dashboard" in local store"

For reference all of my manifests are deployed under the test namespace in K8.

Most of the references I found online talk about ensuring the service has a valid endpoint.

kubectl get endpoints --namespace test

NAME                                       ENDPOINTS                                      AGE
grafana                                    10.68.5.23:3000                                6h4m
hello-kubernetes-second                    10.68.3.9:8080,10.68.5.5:8080,10.68.6.6:8080   6h22m
nginx-ingress-nginx-controller             10.68.5.8:443,10.68.5.8:80                     6h20m
nginx-ingress-nginx-controller-admission   10.68.5.8:8443                                 6h20m

kubectl describe svc grafana -n test


Namespace:         test
Labels:            io.kompose.service=grafana
Annotations:       <none>
Selector:          io.kompose.service=grafana
Type:              ClusterIP
IP:                10.71.248.111
Port:              3000  3000/TCP
TargetPort:        3000/TCP
Endpoints:         10.68.5.23:3000
Session Affinity:  None
Events:            <none>

As far as I can tell, my services do have a valid endpoint. What is it that I'm missing?

csgeek
  • 167
  • 7
  • In your Ingress you are referring to `dashboard` service but in your `service` output you don't have it. It's typo and instead of `dashboard` you should have `grafana` as `serviceName`? If no, please provide `dashboard` service yaml. Error in logs pointing that you dont have `endpoints` for `dashboard` service and I cannot find it in your paste. Please provide also some information about your env (local, cloud), k8s version. – PjoterS Feb 10 '21 at 07:39
  • Sigh. It's always something silly after staring at this for so many damn hours. Thanks! @PjoterS if you want to put your comment as an answer i'll give you the credit for it. – csgeek Feb 10 '21 at 17:37
  • I know this pain :) I'm glad it's working now. – PjoterS Feb 11 '21 at 10:03

1 Answers1

1

As it was mention in comment section, Root cause of the issue was Ingress misconfiguration.

One of the Ingress paths was pointing to non existing service - dashboard. Once it was changed to grafana, ingress should works correctly.

PjoterS
  • 615
  • 3
  • 11