The first thing to be aware of is that modern web browsers may not be the best tool for debugging problems such as these. The browser may even be the cause of them as the server can be configured completely correctly and the issue can be 100% client side.
Contents:
- testing (with
curl and incognito/anonymous browser)
- both tests display the expected content
curl does, but the private browser window does not display the correct content- neither test displays the expected content
testing
Try to not only test with the FQDN http://sub.example.com but use a more complete URL such as http://sub.example.com/some-page.html. When you know that you're behind a proxy, adding additional parameters will usually by-pass cached contents i.e. add and increment a counter such as http://sub.example.com/some-page.html?test=3
test from a "Private/Incognito" browser window.
That reduces the odds of drawing incorrect conclusions based on cached objects.
test from a command line (on a test server) with for example curl -vv http://sub.example.com/some-page.html or use telnet and emulate a web request with
telnet sub.example.com 80
GET /some-page.html HTTP/1.1
Host: sub.example.com
Things to look for in the curl output:
Proxy settings are definitely a danger sign and a real hinderance when debugging server issues. Proxy servers can cache (DNS records and web content), restrict your access and even replace TLS certificates.
curl -vv http://sub.example.com/some-page.html
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,.corp.example.net'
* Uses proxy env variable http_proxy == 'http://[user]:[pass]@proxy.corp.example.net:8080/'
* Trying 10.2.0.80:8080...
^^^ Danger - Proxy server
Test from (test) system that does not rely on a proxy server if you want to properly test the configuration of an internet web server. I usually don't have test systems that will support a fully fledged browser, but I can usually manage a system that will run curl commands from a command line.
curl -vv http://sub.example.com/some-page.html
* About to connect() to sub.example.com port 80 (#0)
* Trying 192.168.2.119...
* Connected to sub.example.com (192.168.2.119) port 80 (#0)
^^^ CHECK is this the correct IP-address
of your webserver?
> GET /some-page.html HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sub.example.com
> Accept: */*
There you observe that a connection gets successfully established to the correct IP-address and port of the web server. The > marker precedes the request headers made by curl.
The server response headers are marked with a < marker and then after an empty line the response body is sent by the server:
< HTTP/1.1 302 Found
| ^^^ The server HTTP STATUS response code
`^^ the HTTP protocol version
< Date: Mon, 07 Feb 2022 13:58:11 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
< Location: https://sub.example.com/some-page.html
< Content-Length: 216
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
...
After the first protocol header HTTP/1.1 comes the HTTP status code the server generates. There a 2xx response means some form of success, 3xx responses are redirects (seen here), 4xx and 5xx are errors.
Of the other headers the server sends the Location: in this example indicates the target of a redirect response.
both tests display the expected content
Conclusion: The web server is configured correctly and the issue is browser related.
The most likely cause: there may have been a permanent redirect configured before the web server configuration was changed and permanent redirects (301) are cached by browsers. Fairly common are cached broken redirects or cached redirects to https and no certificate and https site for sub.example.com is configured yet.
Resolution 1: Clear the browser cache and the site should load as expected.
Resolution 2: Get a SSL certificate for sub.example.com and enable TLS.
Cause 2: Not quite the same root cause, but effectively similar cause and effect to a cached redirect to https, a HTTP Strict Transport Security (HSTS) policy that is set by the example.com domain is also applicable for all (new) subdomains of example.com. Any reasonably modern browser stores and honours HSTS policies and will refuse to connect over plain http to port 80 and will silently rewrite the http://sub.example.com URL to https://sub.example.com and attempt to connect to port 443 with TLS.
Resolution: get a SSL certificate for sub.example.com and enable TLS.
Clearing the HSTS browser cache can work temporarily but that is only effective when you also remove the include subdomains HSTS policy from example.com, which is not recommended.
curl does, but the private browser window does not display the correct content
Conclusion: The web server is configured correctly and the issue is browser or client related.
Cause 1: Not quite the same root cause, but effectively similar cause and effect to a cached HTTP Strict Transport Security (HSTS) policy is a domain on the HSTS pre-load list. But where cached HSTS policies should not be honoured in a "Private/Incognito" windows, the HSTS pre-load list should always apply, even in private/incognito windows.
See this Q&A for an overview on how to determine if either your domain or your TLD is on the HSTS pre-load list:
List of top-level domains (TLDs) that require HTTPS connections, like .dev
Resolution: get a SSL certificate for sub.example.com and enable TLS.
Cause 2: although the DNS records are configured correctly, the client resolver, the desktop where your browser runs, may point to a different IP-address. Check with for example nslookup and dig what the client name server resolves and that do not forget that a hosts file entry for sub.example.com will override DNS.
Resolution: remove the hosts file entry for sub.example.com, or investigate the DNS discrepancy further.
neither test displays the expected content
Conclusion: The issue is server-side, not client-side.
No response
Refer to What causes the 'Connection Refused' message? when it appears that the web server does not respond at all on port 80.
Redirect response
When the web server does respond, take closer look that response:
< HTTP/1.1 302 Found
< Date: Mon, 07 Feb 2022 13:58:11 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
< Location: https://sub.example.com/some-page.html
< Content-Length: 216
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
...
This is a fairly typical response for a web server that is configured to redirect plain HTTP to HTTPS.
The 302 Found status code in the first header is indicative of a temporary redirect. For a permanent redirect that would be a 301 Moved Permanently http status header.
The Location: header show where the client is redirect to.
Cause: when not defined in the web server configuration it is very common for web applications to generate redirects to either https, or a different domain altogether.
On Apache web servers (that allow them) a .htaccess file can also be the source of a redirect.
Resolution: adjust or remove the redirect, or ensure that the redirect becomes valid and something responds on the target of the redirect.
Frequently: get that SSL certificate and enable TLS!
Cause: often the target of the redirect is broken, check for subtle typo's there as well, I have seen missing /'s to redirect to sub.example.comsome-page.html and incorrectly used parameters redirecting to http://some-page.html, or http://wwww.example.com/http://sub.example.com/some-page.html and similar.
Also test for loops by making a request to the Location: multiple levels of redirects usually indicate a broken configuration as well and remove the loop.
Content from different site
When you see content from a different site that you host and not the content for sub.example.com be aware that most web servers will display the content from a default site when they can't match a hostname to a specific site.
Resolution: Re-check your configuration for typo's and confirm that the web server has been restarted with the updated configuration.
