3

I have this website that was working fine until recently. Now users are reporting that it's not opening in their iphones and ipads.

Doesn't matter what browser you try on iOS, it just won't work. Also it doesn't open in Safari when browsed on a Mac machine. Other browsers work fine though (Mac OSX only, not iOS).

Here is the server configuration:

DigitalOcean + Ubuntu 18.04 + Nginx + PHP 7.3 (Laravel) + Let's Encrypt SSL + HTTP2 activated

Before I explain further, I wanna mention that using the same type of configuration as above, I have another server which is running fine and is accessible from all devices. They are almost identical (in terms of config and setup) apart from domain name and ip address obviously.

Another tiny extra info which I think is worth mentioning, is that my users are located in Iran.

Here is a list of things that I've checked:

  • The domain is accessible using ping command from iOS and Mac.
  • The IP address itself is also accessible, and it can be used to browse the site, it will just show a "invalid ssl certificate" warning.
  • The DNS that were used for the domain are all ping-able from iOS and Mac.
  • If a VPN connection is used, the site can be accessed. Which is really weird because everything is technically reachable, the domain, the ip and the dns.
  • Aside from using a VPN, the site can also be accessible if SSL is completely turned off/disabled.
  • Other public websites that are using Let's Encrypt SSL are all accessible. So it can't be a Let's Encrypt issue.

Here is what I've tried so far:

  • Tons of googling. I've even came across with a user with same type of issue on here and here and here
  • I've tried disabling HTTP2.
  • I've double checked my firewall settings, and even tried disabling it.
  • I've tried disabling GZIP.
  • I did a ssl test on ssllabs.com and it shows no problem and it's identical to my working server.
  • Tried using keepalive_disable "safari" in nginx config
  • I've tried swapping ssl_protocols values, like only accepting TLSv1.3 or dropping TLSv1.0
  • Tried using ssl_session_cache
  • Tried changing ssl_ecdh_curve to auto and secp384r1
  • Tried changing ssl_ciphers to HIGH:!aNULL:!MD5; and EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
  • Tried changing ssl_session_tickets to on and off
  • Tried using Strict-Transport-Security
  • Tried commenting /etc/letsencrypt/options-ssl-nginx.conf which is used by certbot
  • Tried different HTTP to HTTPS redirect settings
  • I made sure to use private tab when testing to make sure i'm not looking at a sort of faulty cached attempt to reach the site.
  • I did service nginx reload & restart everytime I made a change to config files.
  • I did lots of sudo reboots to make sure it's not a simple reboot issue.

As a final resort, I even went through the trouble of reinstalling the OS from scratch and setting up the server again. Still Doesn't work. And no I didn't copy my config files, I did every command manually and modified config files carefully to make sure I don't bring any problems from previous setup. This also means I did a fresh SSL request from Let's Encrypt (using cerbot). Although I don't know if they generate a new one or the send you the one from last time.

EDIT 1: I want to throw a random thought in here. I really don't know about advanced networking but maybe something from Iran censorship firewalls is meddling with the SSL connection and that causes Apple devices to fail establishing the connection. I heard Apple is strict in it's own way and requires a specific SSL connection unlike chrome and firefox. those 2 might ignore a slight error but Apple devices don't.

EDIT 2: I checked the Safari while Wireshark is tapping. It seems only few packets gets exchanged only no matter what I do. Also it seems Safari is using TLSv1.0 even though I've disabled it in nginx config. I really don't know how completely turn it off so Safari won't even try to use it to communicate with the server.

EDIT 3: I tried using a different SSL (using comodo's 3 month free ssl trial program) and the issue is still there... I read some people saying they resolved some similar issues after changing their SSL. I don't know why it's not working for me.

EDIT 4: I decided to change DNS servers of the domain to see if it's a DNS issue. I don't know why and how but after I changed them, Safari could load the site briefly. But after sometime it stopped working again.

EDIT 5: No luck with disabling external script codes in my site, like google analytics ( because their service is kinda blocked for Iranians ). Again, I read people reports implying that some fixed their issue by disabling external scripts.

EDIT 6: I'm starting to believe this is not a networking or server config issue on my part. It really looks like a 3rd party interfering with the request. I don't know about Apple and how it handles networking, but I think Apple connection/packeting negotiations is causing some sort of red flag so to speak and that makes Iranian censorship/firewall to drop the client connection.

EDIT 7: I even changed the server's datacenter, with a fresh new ip address. the issue still exists :(

EDIT 8: This is the /var/log/nginx/error.log output when it's set to debug. It shows these lines when I initiate a request from a Safari client.

nginx error.log output

I did try to disable php fpm to make sure it's not a php bottleneck issue. Also I tried to tweak some random parameters, like bumping net.core.somaxconn to 1024 or increasing backlog in php fpm config files. Also when I watch the system using htop everythink looks normal, no cpu usage spikes and no specific process jumping to top of the list.

EDIT 9: I swapped Nginx to Apache2 to check if it's a webserver issue. Well, it wasn't.

xperator
  • 437
  • 2
  • 11
  • 23
  • In HTTP it work? – yagmoth555 Jul 20 '19 at 11:11
  • @yagmoth555 yes it works without any issues if I disable ssl. – xperator Jul 20 '19 at 14:40
  • "...users are reporting that it's not opening..." Ok but is it all working for you? – Rob Jul 20 '19 at 16:19
  • @Rob No it's not working for anyone who's trying to accessing from Iran, including me. I stated it that way meaning that is how I got informed. – xperator Jul 20 '19 at 16:21
  • I assumed you were not in Iran. Can anyone outside of that area access it? – Rob Jul 20 '19 at 16:22
  • 1
    @Rob yes. and as I mentioned in the main post, using a VPN would solve the problem. I think something from Apple's devices is causing Iranian firewall/censorship devices to drop the connection. But it's probably a really rare case. Because all other sites and services work just fine. – xperator Jul 20 '19 at 16:29
  • It's maybe a bit late now but in case anyone else comes across this, I wonder if enabling other TLS versions or ciphers would have worked. – LTPCGO Sep 17 '19 at 02:28
  • @LTPCGO if you read the post you could see I've already tried changing TLS versions and ciphers, it just doesn't work. You better off moving to a different host than wasting your time on this matter like how i did – xperator Sep 17 '19 at 06:28
  • Somehow I missed that, I only caught where you mentioned V1.0! Meaculpa – LTPCGO Sep 17 '19 at 06:43
  • Hello! Sorry, but have you found a solution? Because I have the same problem :( – nabiullinas Oct 19 '20 at 07:47
  • @nabiullinas Unfortunately no and I no longer work on that project which I had the issue. Back then when I talked to a networking expert they said it's probably a false positive caused by the government firewalls (countries that don't like encrypted data) – xperator Oct 19 '20 at 11:08

4 Answers4

1

Since you doubt the proxy in-between you and the website, can you bypass it by using the -L option in ssh? It will provide a secure tunnel between you and your website that cannot be altered by the proxy.

For example, execute this on your desktop: ssh -L 2222:127.0.0.1:443 user@mywebsite.com

Once this command has logged into your server, fire up the browser on the same desktop to https://127.0.0.1:2222

If you now see you site and its SSL certificate, everything is setup correctly - theres someone in the middle altering your SSL connection when you are not using the tunnel.

Austin Dixon
  • 39
  • 3
  • I don't have any problem SSH'ing into the vps and yeah i know someone, or should i say, the government firewall is killing the apple devices requests for some reason. I know they're sensitive to encrypted data. But I really don't know why and how only apple devices are affected. they're triggering some sort of protocol or specific routing that the censorship hardwares don't like. I really wish I could fix this at user's level. This is so randomly only happening to my site. It used to work fine few weeks ago and now it's just getting picked up by the firewall. ofc I can disable SSL & keep going – xperator Jul 25 '19 at 20:21
  • OpenSSL includes a client that you can test with. For example: openssl s_client -connect mywebsite.com:443 Does this connect? – Austin Dixon Jul 25 '19 at 20:47
  • It does connect. I did mention in my post that only the browsers are affected. command line connections work fine. – xperator Jul 26 '19 at 09:24
1

I found a temporary solution to my problem. I just moved the server to a different hosting/datacenter!

Update: Actually the problem reappeared again after few months... I really don't know what technical issue is causing the problem or how to fix it.

Update 2: I talked to an IT networking expert and they told me the issue actually relies in government firewall rules picking up on my domain name ( false/wrong censorship ) and it doesn't matter what webserver configuration or webhosting I use, it just keeps getting blocked by firewalls out of my reach.

xperator
  • 437
  • 2
  • 11
  • 23
  • 1. did you check your firewall using the ufw?, 2. did you add the auto renewal of certbot certificate in your cron using '0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && /usr/local/bin/certbot-auto renew -q" | sudo tee -a /etc/crontab > /dev/null' this comment? 3. get the ssl params settings from my answer https://stackoverflow.com/a/63750775/3462686. 4. And also check the folder permission and chmod – Karthikeyan Ganesan Sep 05 '20 at 05:41
0

It could just be the certificate, do you have SubjectAlternativeName properly set? Only using the SubjectName for the DNS name is deprecated. Do you use certificate pinning, OSCP stapling, HSTP, and other relatively new TLS settings? That might cause issues on regulated networks.

Some internet users need to use a proxy with SSL-inspection. Many corporate environments use products similar to IronPort or Bluecoat to detect covert channels and malware as part of their security policy. The way to do this originated from an article in the underground magazine Phrack 76. It boils down to a simple setup where each client has an additional CA root certificate of the proxy that it trusts, the proxy in turn re-initiates connections on behalf of each client, "opening the SSL" by MIT, countries like Iran and China monitor the internet at large. Due to this there is no PFS, and several things might fail that depend on the client checking the servers' crypto, because it is altered or ssl is even completely removed. If you don't care about it then you can downgrade your server certificate settings to "export quality" by removing mentioned options.

bbaassssiiee
  • 142
  • 6
  • I don't know about about "SubjectAlternativeName" but I tried Let's Encrypt SSL and as I mentioned later in my post in my 3rd edit, I swapped to Comodo's free 3month trial to check if it's a Let's Encrypt issue, and it wasn't. Also I tried every possible combination of ssl options like the ones you mentioned, OSCP stapling, HSTP, etc... – xperator Jul 22 '19 at 19:09
  • The idea is to use a little SSL as possible, to allow SSL inspection. You tend to tighten it too much. – bbaassssiiee Jul 22 '19 at 19:27
0

@austin-dixon's answer is on the right track. That test would have to be done from the same country the users are in, which may not be an option if you are not already in that country.

You may be able to at least validate the configuration by using the Qualys SSL test at https://www.ssllabs.com/ssltest/. The letter score doesn't matter in this case, just scroll down to Handshake Simulation and see how the Apple devices are expected to perform under normal circumstances.

Either way that just confirms what you already suspect, that there's something in between those users and the site. There are not a lot of ways for a developer to get around that situation short of disabling HTTPS completely. It's possible the man in the middle requires some of the weaker cipher suites or lower TLS versions than the server is configured for, but I wouldn't know which ones to suggest in that case.

wrk2bike
  • 46
  • 3
  • You should have read the post and it's comments. I did state that I'm located in the problematic country at the moment. Also I did run a ssl test on ssllabs.com it shows no errors or problems. and the apple devices are looking fine under handshake simulation. And I did try to change the ciphers and the TLS version. It didn't help the situation. – xperator Jul 26 '19 at 09:31
  • Did @austin-dixon's test work? You said SSH works but not whether browser traffic going through that SSH tunnel shows the problem behavior. – wrk2bike Jul 26 '19 at 14:19