Caddy reverse proxy curl works internally, but externally returns content-length: 0

Question

Background and Problem

I'm trying to set up Caddy as a reverse proxy between two other web applications and a static file server (all on one machine). When I curl the internal IP, it works as expected, but when I try to curl the external IP, it returns content-length: 0. Ultimately, my question is why is this happening?

I'm not using a domain name, just straight IP. I know I could get a free domain from a bunch of different DNS hosts; I'd really rather not.

Network Setup

I have a router connected to a single single physical server that is home to Caddy and the two web applications. I'm forwarding port 443 from my router to the server where Caddy is installed.

`curl` Results

192.168.1.5 is my internal IP. 203.0.113.0 means my external IP (thanks @Nikita Kipriyanov for pointing out RFC1918 and RFC5737).

It looks like both requests are making it to Caddy at least (based on the HTTP/2 200 line), but I don't understand why when the request comes from the external IP, it returns nothing. Neither request generates any errors in the Caddy terminal output.

Like I mentioned before, I'm not using a domain name, just straight IP. I'm not sure if that has anything to do with the problems I'm seeing, but it seems to me that it shouldn't matter; I've already told curl to ignore the warnings about untrusted certs with the -k option.

Output when using internal IP

admin@server:~$ curl -vk https://192.168.1.5/
*   Trying 192.168.1.5:443...
* TCP_NODELAY set
* Connected to 192.168.1.5 (192.168.1.5) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Aug 21 00:46:28 2022 GMT
*  expire date: Aug 21 12:46:28 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561a91115210)
> GET / HTTP/2
> Host: 192.168.1.5
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< content-type: text/html; charset=utf-8
< cross-origin-opener-policy: same-origin
< referrer-policy: same-origin
< server: Caddy
< x-content-type-options: nosniff
< x-frame-options: DENY
< content-length: 2921
< date: Sun, 21 Aug 2022 05:07:17 GMT
 
<!doctype html>
<html lang="en">
  <body>
    <h1>Hello World!</h1>
  </body>
</html>
* Connection #0 to host 192.168.1.5 left intact

Output when using external IP

admin@server:~$ curl -vk https://203.0.113.0/
*   Trying 203.0.113.0:443...
* TCP_NODELAY set
* Connected to 203.0.113.0 (203.0.113.0) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Aug 21 00:46:28 2022 GMT
*  expire date: Aug 21 12:46:28 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e3ae7aa210)
> GET / HTTP/2
> Host: 203.0.113.0
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< server: Caddy
< content-length: 0
< date: Sun, 21 Aug 2022 05:08:16 GMT
< 
* Connection #0 to host 203.0.113.0 left intact

Caddyfile

192.168.1.5 is my internal IP.

I know app-one has two handles; there's a configuration issue where the app-one drops the trailing / and redirects, which then causes app-two (the one with the catch-all handle) to match, so the fix was to create a second handle just for that case. I'm pretty confident that's not the problem.

{
        default_sni 192.168.1.5
}

https://192.168.1.5:443 {
        handle /file-server/* {
                root * /var/
                file_server browse
        }

        handle /app-one/* {
                reverse_proxy /app-one/* localhost:30000
        }

        handle /app-one {
                reverse_proxy /app-one localhost:30000
        }

        handle {
                reverse_proxy * localhost:8000
        }
}

Don't obfuscate RFC1918 IPs, they tell nothing about you anyway, but the details about your network they may show almost always required to understand the details of the problem. Present them as is, it is safe. You may want to hide your public IPs, but only replace them with RFC5737 addresses in that case and never invent addresses you don't own. // I *suspect* you are listening on wrong IP, but your quesiton is formed in a way I can't be sure. Are both "xxx..." represent the same address? Does this address belong to the system where Caddy is running? Is it the public address or private? — Nikita Kipriyanov, Aug 21 '22 at 06:21
Thanks @NikitaKipriyanov. I've updated the post based on your suggestions, and yes, all the `xxx.xxx.xxx.xxx` represented `192.168.1.5` which is the address to the system that Caddy is running on. It is private, but I'm forwarding `203.0.113.0:443` to `192.168.1.5:443`. — claypooj, Aug 21 '22 at 18:08
Are you trying to access an external address from within the same network 192.168.1.0/24 where actual server is deployed? — Nikita Kipriyanov, Aug 22 '22 at 04:55
I have tried both from a machine on the internal network (192.168.1.16) and had a friend try from one on an external network (not sure what the IP is for the external one, but it's a thousand miles away); same result. — claypooj, Aug 22 '22 at 21:13

score 1 · Answer 1 · answered Aug 23 '22 at 19:50

The answer from Nikita Kipriyanov was right; I'm just adding this to show what my final configuration looked liked.

Explanation

As Nikita said:

... in the second transcript the Host is different:
GET / HTTP/2
Host: 203.0.113.0
and there is nothing in the Caddyfile to match it. So the behavior is expected.

It turns out the problem was in my Caddyfile. I had improperly configured my site address; Caddy was returning a response with no content because it wasn't matching any routes provided. The final structure of my file looks like this:

{
        default_sni 192.168.1.5
}

https://203.0.113.0:443, https://192.168.1.5:443 {

        handle /static/* {
                root * /srv
                file_server browse
        }

        handle /foundry/* {
                reverse_proxy /foundry/* localhost:30000
        }

        handle /foundry {
                reverse_proxy /foundry localhost:30000
        }

        handle {
                reverse_proxy * localhost:8000
        }
}

A few important things I learned:

The request forwarded from the router does not change the Host header to the internal IP address; it just forwards the whole request to the internal IP address. So instead of receiving a request for the route https://192.168.1.5, Caddy was actually receiving a request for the route https://203.0.113.0. Since there wasn't a route for https://203.0.113.0, it was just sending back a blank page.
The router was configured to listen on port :580 and forward requests to 192.168.1.5:443. I had overlooked this detail in my original question. Something important to note is that the port on the site address to match is not the external port open on the router; it's the port that Caddy is listening to on the machine where Caddy is running. This tripped me up because I'm forwarding port 580 from the router to port 443 on the machine, but in order for Caddy to match the route, I couldn't use https://203.0.113.0:580 (which I had tried as a route during earlier troubleshooting); it has to be https://203.0.113.0:443.

Bottom Line

If you're using IP addresses instead of domain names, make sure your routes for external IPs follow this form:

<http | https>://<external IP>:<port that Caddy is listening on>

*request forwarded from the router does not change the Host header to the internal IP address* --- in case of HTTPS it cannot, the traffic looks like garbage to the router — Nikita Kipriyanov, Aug 24 '22 at 05:19

Nikita Kipriyanov · Accepted Answer · 2022-08-23T11:27:29.367

I believe this is due to how address matcher works. Notice that in the first transcript you have (in the HTTP request):

GET / HTTP/2
Host: 192.168.1.5

which is successfully matched with https://192.168.1.5:443 of the Caddyfile, while in the second transcript the Host is different:

GET / HTTP/2
Host: 203.0.113.0

and there is nothing in the Caddyfile to match it. So the behavior is expected.

You can check this by setting arbitrary Host header to 192.168.1.5 with Curl:

curl -H "Host: 192.168.1.5" -vk https://203.0.113.0/

This could work from outside, but there also could be problem with SNI hostname, which will be still set to 203.0.113.0. I am not sure, but adding curl -vk --resolve 192.168.1.5:443:203.0.113.0 https://192.168.1.5/ may help in this case (it is designed to work with host name, like in curl --resolve example.com:443:127.0.0.1 https://example.com/...; I don't know whether it is able to "resolve" IP address to another IP address).

How to fix this? You can try to add other host match to this block:

https://192.168.1.5:443, 203.0.113.0 { ... }

I didn't checked this, but I believe the first one will set the binding, and second is there only to match a Host header.

However, I think better is to never access HTTPS servers by raw IP addresses. Use names.