Istio egress gateway HANDSHAKE_FAILURE_ON_CLIENT_HELLO with custom certs

Question

What we ware trying to achieve is point mesh traffic to an external service via an egressgateway.

We tried several iterations, and now trying with an egressgateway in between.

The external service is running with our certificates.

Mesh > Egressgateway > External service.

The error on the gateway is:

[2020-08-21T14:52:37.523Z] "GET / HTTP/1.1" 503 UF,URX "-" "TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE 268435610:SSL routines:OPENSSL_internal:HANDSHAKE_FAILURE_ON_CLIENT_HELLO" 0 91 62 - "10.32.64.16" "curl/7.61.1" "f57b76e9-99b1-43bd-8905-1226ab2c7e69" "<vm-name>.europe-west1-b.c.<google-project-name>.internal" "10.32.3.207:8123" outbound|8123|notebook-ext|<vm-name>.europe-west1-b.c.<google-project-name>.internal - 10.32.65.2:8123 10.32.64.16:54912 <vm-name>.europe-west1-b.c.<google-project-name>.internal -

Yamls:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: notebook
  namespace: istio-system
spec:
  hosts:
  - <vm-name>.europe-west1-b.c.<google-project-name>.internal
  ports:
  - number: 80
    name: http-port-for-tls-origination
    protocol: http
  - number: 8123
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway-notebook
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 80
      name: http-port-for-tls-origination
      protocol: HTTP
    hosts:
    - <vm-name>.europe-west1-b.c.<google-project-name>.internal
  - port:
      number: 8123
      name: https
      protocol: HTTPS
    hosts:
    - <vm-name>.europe-west1-b.c.<google-project-name>.internal
    tls:
      mode: MUTUAL
      serverCertificate: /temp-certs/cert-chain.pem
      privateKey: /temp-certs/key.pem
      caCertificates: /temp-certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-notebook
  namespace: istio-system
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: notebook
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 8123
        tls:
          mode: ISTIO_MUTUAL
          sni: <vm-name>.europe-west1-b.c.<google-project-name>.internal
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-notebook-through-egress
  namespace: istio-system
spec:
  hosts:
  - <vm-name>.europe-west1-b.c.<google-project-name>.internal
  gateways:
  - mesh
  - istio-egressgateway-notebook
  http:
  - match:
    - gateways:
      - mesh
      port: 80
      sniHosts:
      - <vm-name>.europe-west1-b.c.<google-project-name>.internal
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: notebook
        port:
          number: 8123
  - match:
    - gateways:
      - istio-egressgateway-notebook
      port: 8123
      sniHosts:
      - <vm-name>.europe-west1-b.c.<google-project-name>.internal
    route:
    - destination:
        host: <vm-name>.europe-west1-b.c.<google-project-name>.internal
        subset: notebook-ext
        port:
          number: 8123
        weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: notebook
  namespace: istio-system
spec:
  #exportTo: ["."]
  host: <vm-name>.europe-west1-b.c.<google-project-name>.internal
  subsets:
  - name: notebook-ext  
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 8123
      tls:
        mode: SIMPLE
        sni: <vm-name>.europe-west1-b.c.<google-project-name>.internal
        serverCertificate: /temp-certs/cert-chain.pem
        privateKey: /temp-certs/key.pem
        caCertificates: /temp-certs/root-cert.pem

The end goal for this is to have something available in the mesh on http which will go via the egressgateway and the TLS will originate from there. This is because we want to direct traffic from a public ingressgateway back out of the mesh to the external service via the egressgateway. To have HTTPS

The certs above are present in the egress gateway pod as per curl below:

If I exec in the egressgateway pod and curl https://<vm-name>.europe-west1-b.c.<google-project-name>.internal:8123 --cacert /temp-certs/root-cert.pem -v i get the response from the external VM.

Any ideas why this would be happening?