ingress nginx upstream sent no valid HTTP/1.0 header while reading response header from upstream

Question

I'm trying to setup an nginx ingress controller for services in my namespace. One of the backend services accept HTTP traffic on port 80, the other accepts only HTTPS traffic on port 443. See the description of those both services

$ kubectl describe svc service-1 -n monit
Name:              service-1
Namespace:         monit
Labels:            app=service-1
Annotations:       <none>
Selector:          app=service-1
Type:              ClusterIP
IP:                10.104.185.173
Port:              https  443/TCP
TargetPort:        8443/TCP
Endpoints:         10.1.0.95:8443
Session Affinity:  None
Events:            <none>

$ kubectl describe svc service-2 -n monit
Name:              service-2
Namespace:         monit
Labels:            app=service-2
Annotations:       <none>
Selector:          app=service-2
Type:              ClusterIP
IP:                10.110.93.64
Port:              service  80/TCP
TargetPort:        3000/TCP
Endpoints:         10.1.0.87:3000
Session Affinity:  None
Events:            <none>

Here is my ingress configuration

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-monit
spec:
  rules:
  - host: service-2.localhost
    http:
      paths:
      - path: /
        backend:
          serviceName: service-2
          servicePort: 80
  - host: service-1.localhost
    http:
      paths:
      - path: /
        backend:
          serviceName: service-1
          servicePort: 443

When I look into the Nginx configuration things look OK

$ kubectl describe ingress ingress-monit -n monit                  
Name:             ingress-monit
Namespace:        monit
Address:          localhost
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  service-2.localhost               
                                  /   service-2:80 (10.1.0.87:3000)
  service-1.localhost  
                                  /   service-1:443 (10.1.0.95:8443)
Annotations:                      Events:
  Type                            Reason  Age   From                      Message
  ----                            ------  ----  ----                      -------
  Normal                          CREATE  31m   nginx-ingress-controller  Ingress monit/ingress-monit
  Normal                          UPDATE  30m   nginx-ingress-controller  Ingress monit/ingress-monit

Now the problem is that I can access properly my service-2, with http://service-2.localhost/ but I cannot access service-1. Visiting http://service-1.localhost/ on chrome gives me

This site can’t be reachedThe webpage at https://service-1.localhost/ might be temporarily down or it may have moved permanently to a new web address.
ERR_INVALID_RESPONSE

When I look into Nginx logs, I see:

$ kubectl logs -n monit ingress-nginx-controller-bbdc786b4-8crdm -f
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.32.0
  Build:         git-446845114
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.17.10

-------------------------------------------------------------------------------
. . .
2020/06/02 22:56:47 [error] 2363#2363: *64928 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 192.168.65.3, server: service-1.localhost, request: "GET / HTTP/1.1", upstream: "http://10.1.0.95:8443/", host: "service-1.localhost"
192.168.65.3 - - [02/Jun/2020:22:58:13 +0000] "GET / HTTP/1.1" 200 7817 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 594 0.005 [monit-service-2-80] [] 10.1.0.87:3000 30520 0.005 200 2baefff713047b14a81643650cb50c4c

The error seems to be related to the service-1 returning bad response upstream sent no valid HTTP/1.0 header while reading response header from upstream. The thing is if I use kubectl proxy I can properly access that service!!

Any ideas how I could figure out what's the real issue??

score 3 · Accepted Answer · answered Jun 03 '20 at 04:14

3

You'll need the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation on that Ingress which connects upstream via SSL, since by default the ingress controller assumes all in-cluster upstreams use HTTP

You cannot, that I'm aware of, add that annotation per rule: so you'll have to create a second Ingress just for it:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-monit-service-1
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: service-1.localhost
    http:
      paths:
      - path: /
        backend:
          serviceName: service-1
          servicePort: 443

answered Jun 03 '20 at 04:14

mdaniel

2,338
1
8
13

Thanks for the answer, is it too much overhead having two ingress controllers? would it be better if I make all internal traffic on HTTP then enable TLS on the ingress so that I would have HTTPS with the client? – bachr Jun 03 '20 at 04:37
1

I'm not aware of any overhead from multiple Ingresses other than the size of the generated `nginx.conf` grows larger. There's no "better," only choices with tradeoffs. It depends on what purpose having TLS inside the cluster does for your situation – mdaniel Jun 03 '20 at 04:44
1

@bachr also, remember that an Ingress object (kind: Ingress) is a set of rules, you can create all the ingresses rules you need, in the end the Ingress Controller will summarize these rules. Having two Ingress Controllers is possible, but more complex and usually used for separate objectives. – Will R.O.F. Jun 03 '20 at 14:11
1

thanks @willrof now I understand, I was thinking an ingress is a service that may take resources (cpu/mem), if it is rule then the proposed solution makes sense. – bachr Jun 03 '20 at 15:48

ingress nginx upstream sent no valid HTTP/1.0 header while reading response header from upstream

1 Answers1