1

I'm using NSX/NCP Ingress in a dedicated VMWare PKS cluster. I am attempting to stand up ingress to Elasticsearch and the backend service uses HTTPS. I can not seem to find a way to have NSX Ingress talk to a backend HTTPS service.

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.0/ncp-kubernetes/GUID-E03D6EE5-9C6C-457F-AD81-25CF2056F4D8.html

The load balancer will terminate TLS and send HTTP requests to the default backend server if there is a TLS Ingress (in the cluster for the Kubernetes/PKS use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.

Background:

  1. In this environment, a k8s PKS cluster has no control over DNS. One IP is allocated per dedicated cluster and a wildcard DNS record is pointed to that IP.
  2. Only the NSX class Ingress controller gets to use that dedicated IP (kuberneties.io/ingress.class: nsx)
  3. NSX Ingress controllers do not support the ingress.kubernetes.io/secure-backends: 'true' annotation.
  4. I can deploy other ingress controllers. For instance, I stood up an nginx ingress controller with the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation and that works. However, these ingress controllers get a randomly assigned IP (which may change) and no DNS records.
  5. I have tried removing the tls section and setting the ncp/ssl-mode: reencrypt annotation and only get a 502 error. Would be willing to revisit this.

What I'm looking for:

  1. Ideally, a way I missed to just have NSX ingress work with a secure backend.
  2. Failing that, a way to make the ncp/ssl-mode: reencrypt option work for this scenario case. I don't want to set a default ingress (no host in the rule)
  3. Failing that, the simplest solution to present the eventsink-opendistro-es-client-service port 9200 HTTPS as an HTTP service to an NSX ingress controller. I've considered standing up a separate nginx pod that connects to the service on the backend and presents it as HTTP on the frontend, which the ingress controller can then hit.

I also don't want to rebuild the Elasticsearch image to a non-https custom one, but if there's a helm chart method to make the exposed port 9200 HTTP only, I can do that.

Finally, here is my ingress definition as it stands:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: eventsink-opendistro-es-client
  annotations: 
    kubernetes.io/ingress.class: nsx
    ingress.kubernetes.io/secure-backends: 'true'
    # ncp/ssl-mode: reencrypt
  namespace: default
spec:
  rules:
  - host: elasticsearch.clustername.pks.example.net
    http:
      paths:
      - backend:
          serviceName: eventsink-opendistro-es-client-service
          servicePort: 9200
  tls:
  - hosts:
    - elasticsearch.clustername.pks.example.net
    secretName: clustername.pks.example.net-ssl
ytjohn
  • 417
  • 5
  • 11

1 Answers1

1

The workaround I ended up using was to create a reverseproxy service to expose 9200 HTTPS as HTTP to the ingress controller.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: eventsink-reverseproxy-conf
data:
  default.conf: |
    server {
        listen       80;
        server_name  _;
        location / {
            proxy_pass    https://eventsink-opendistro-es-client-service:9200;
        }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eventsink-reverseproxy
  labels:
    app: eventsink-reverseproxy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: eventsink-reverseproxy
  template:
    metadata:
      labels:
        app: eventsink-reverseproxy
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.0
        ports:
        - name: http
          containerPort: 80
        volumeMounts:
              - name: nginxconfd
                mountPath: /etc/nginx/conf.d
      volumes:
        - name: nginxconfd
          configMap:
            name: eventsink-reverseproxy-conf

---
apiVersion: v1
kind: Service
metadata:
  name: eventsink-reverseproxy-svc
spec:
  selector:
    app: eventsink-reverseproxy
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
ytjohn
  • 417
  • 5
  • 11