ingress returns 502 / TLS backend communication issue

Question

i am operating a k8s cluster with cert-managerto obtain letsencrypt certs. I just set up nifi (https://github.com/cetic/helm-nifi) within this cluster, where the workload exposes a TLS secured port which is abstracted by a respective service.

i set up an ingress which routes the traffic to the respective nifi service. The certs is properly obtained as expected, but i keep getting a 502 bad gatewaymessage when trying ot access the service via ingress.

The ingress controller reveals the reason:

2020/11/07 15:45:20 [error] 10687#10687: *154495249 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 134.3.215.120, server: https://[my-domain], request: "GET /favicon.ico HTTP/2.0", upstream: "https://10.42.1.137:8443//favicon.ico", host: "https://[my-domain]", referrer: "https://[my-domain]"

this is my ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
 
    ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/secure-backends: "true"
  generation: 1
  labels:
    app: nifi
    chart: nifi-0.5.5
    heritage: Helm
    release: nifi
  name: nifi-ingress
  namespace: nifi
  resourceVersion: "138180326"
  uid: a6b561a8-1fc0-11eb-b2e1-6a9097f8076d
spec:
  rules:
  - host: [my-domain]
    http:
      paths:
      - backend:
          serviceName: nifi
          servicePort: 5422
        path: /
  tls:
  - hosts:
    - [my-domain]
    secretName: nifi-tls

it appears that ingress wants to establish a TLS connection which is rejected by the pod (quite obviously), but i have no idea how to fix this. It seems apparent that installing e.g. a self-signed cert on the nifi pod would do the job, but I have no idea how to do that and I rather believe i am doing something wrong here since this should work out of the box.

Thankful for any hint or idea.

UPDATE / clarification:

I set it up with a standard configuration now to make things a bit easier to understand. The core issue is from the ingress controller log:

2020/11/18 23:01:25 [error] 15529#15529: *166867031 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: xxx, server: [nifi-ingress-hostname], request: "GET /favicon.ico HTTP/2.0", upstream: "https://[nifi-pod]:8443/favicon.ico", host: "[nifi-hostname]", referrer: "https://[nifi-ingress-hostname]"

[nifi-ingress-hostname] is the ingress which routes traffic to the nifi service on port 8443. The service forwards the traffic to [nifi-pod]:8443. the service on [nifi-pod] listens on 8443 for encrypted communication.

However the issue seems to be that the last step in the communication ingress to pod does not work since the ssl handshake fails. I added those annotations to the ingress:

    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

to pass the encrypted traffic to the pod directly, but this had no effect. Somehow it seems that either i have to change the helm chart to change the service on the pod to http rather than https or ensure an encrypted connection with self-signed certs or am i missing anything?

`nginx.ingress.kubernetes.io/backend-protocol: HTTPS` is the annotation that describes the protocol nginx will use with the upstream Pod; if your upstream isn't listening on port 5422 for TLS traffic, you should remove that annotation (since the default is `HTTP`) — mdaniel, Nov 08 '20 at 03:11
correct, but this is essentially the problem, since the pod is listening for TLS traffic, yet since the ingress handles the SSL endpoint termination and the pod requires TLS, yet there is no TLS connection possible between the ingress controller and the pod. So the issue seems to be that taffic between ingress controller and pod should be secured by a self signed certificate, which seems to be hard to handle. — michafn, Nov 08 '20 at 11:58
*yet there is no TLS connection possible between the ingress controller and the pod* that's the purpose of that annotation, and I think I got confused because the first paragraph says "TLS secured port" and the last paragraph says "would do the job" as if port 5422 is not **currently** serving TLS. If you want help, you'll have to tighten up your question to show what, exactly, is in place. You can do some of your own troubleshooting by issuing `curl -vk https://nifi.nifi.svc.cluster.local:5422` from a Pod and seeing what it returns — mdaniel, Nov 08 '20 at 18:45
Related to that "tightening up your question" observation, your error message shows `https://10.42.1.137:8443//` but there's no way for us to know what `10.42.1.137` corresponds to, and it is highly suspicious that the URL is `8443` when your nifi is `5422` — mdaniel, Nov 08 '20 at 18:47
you are right, that was a bit misleading. I added some more information above. Appreciate your help! — michafn, Nov 18 '20 at 23:26

ingress returns 502 / TLS backend communication issue

0 Answers0