Istio : HTTPS Traffic converted to HTTP with port set as 443

Question

Bug description

We have setup an istio over on eks cluster & a java app is hosted in it. The pod has been created along with service with type ClusterIP

We have created Virtual Service, Gateway & set the istio ingress gateway as a NodePort. In front of the istio ingress gateway, we placed the AWS Application Load Balancer. We created a route53(DNS) entry which points to the above said ALB

Now, all the services & pods are UP

Now, on hitting the DNS with https, the request info we get in the above said Spring Boot Java Application, has been changed from https to http with port 443

For Eg: if I make a curl request to say curl https://sarath.somedomain.com/helloworld, in our java app, while we get the request info we get them as http://sarath.somedomain.com:443/helloworld

Also we have printed the headers like x-forwarded-proto, x-forwarded-port in our Java App.

x-forwarded-proto value we get in the java app is http x-forwarded-port value we get in the java app is 443 x-forwarded-for value we get in the java app is null

Our key suspicion is with envoy which does this protocol conversion from https to http

Expected behavior

If I hit say curl https://sarath.somedomain.com/helloworld, in our java app we should get the request info as https://sarath.somedomain.com/helloworld

Steps to reproduce the bug

Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: sarath-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "sarath.somedomain.com"
    - "prasath.somedomain.com"

Virtual Service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: sp-vs
spec:
  hosts:
  - "sarath.somedomain.com"
  gateways:
  - sarath-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: javaapp-svc

ALB Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: sarathingress
  namespace: istio-system
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/subnets: subnet-XXXXXXXX,subnet-XXXXXXXX
    alb.ingress.kubernetes.io/security-groups: sg-XXXXXXXX
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:XXXXXXXX:certificate/XXXXXXXX
spec:
  rules:
    - host: sarath.somedomain.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: istio-ingressgateway
              servicePort: 80

App Service:

apiVersion: v1
kind: Service
metadata:
  name: javaapp-svc
  labels:
    app: javaapp-svc
    name: javaapp-svc
spec:
  selector:
    app: javaapp-svc
  ports:
  - name: http
    port: 80
    targetPort: 9090
    protocol: TCP

Version (include the output of istioctl version --remote and kubectl version)

Output of istioctl version --remote:

client version: 1.2.5 citadel version: 1.2.5 galley version: 1.2.5 ingressgateway version: 1.2.5 pilot version: 1.2.5 policy version: 1.2.5 sidecar-injector version: 1.2.5 telemetry version: 1.2.5

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-29T16:15:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-eks-5ac0f1", GitCommit:"5ac0f1d9ab2c254ea2b0ce3534fd72932094c6e1", GitTreeState:"clean", BuildDate:"2019-08-20T22:39:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

Istio is installed through helm, below is the command used:

helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system
helm template \
  --set gateways.istio-ingressgateway.type=NodePort \
  --set prometheus.enabled=true \
  --set grafana.enabled=true \
  --set tracing.enabled=true \
  --set kiali.enabled=true \
  --set "kiali.dashboard.jaegerURL=http://jaeger-query:16686" \
  --set "kiali.dashboard.grafanaURL=http://grafana:3000" \
   install/kubernetes/helm/istio \
  --name istio --namespace istio-system > $HOME/istio.yaml

Environment where bug was observed (cloud vendor, OS, etc)

AWS, EC2 Machine, Amazon Linux v2

Looks like he you are missing one [annotation](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/tasks/ssl_redirect/#example-ingress-manifest), that will force http -> https redirection "the ssl-redirect action must be first rule(which will be evaluated first by ALB)" — Wytrzymały Wiktor, Sep 23 '19 at 13:25
The quickest way to achieve it would be via setting servicePort: 443 in the backend configuration of sarathingress Istio backend. — Wytrzymały Wiktor, Sep 23 '19 at 13:25

score 0 · Answer 1 · answered Mar 28 '22 at 16:20

Have you tried updating the VirtualService to set X-Forwarded-Proto?, like below:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: sp-vs
spec:
  hosts:
  - "sarath.somedomain.com"
  gateways:
  - sarath-gateway
  http:
    - headers:
        request:
          set:
            X-Forwarded-Proto: https
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: javaapp-svc

I believe this should give you the behavior you expect

Istio : HTTPS Traffic converted to HTTP with port set as 443

1 Answers1