1

After a PC reconfiguration I am unable to use Docker properly, since some curl commands are rejected due to SSL/TLS issues.

In just one example curl -vfsSL https://apt.releases.hashicorp.com/gpg returns the following error:

*   Trying 52.222.214.125:443...
* TCP_NODELAY set
* Connected to apt.releases.hashicorp.com (52.222.214.125) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

After some digging, I now now know that this issue also occurs within my WSL image, but not on host Windows OS. Hence, I believe this must be an issue that originates with my WSL setup, and not caused by Docker itself (?).

There are quite a few related questions on serverfault/stackoverflow but no solutions I found really apply to this case:

FWIW I work at an enterprise, with IT-issued OS. Obviously that could be a source of error, but they are unable to help me debug this issue. One a colleague's PC, however, it works flawlessly.

Any ideas?

PC Setup:

  • Windows 10 Enterprise
    • Version: 21H1
    • OS build: 19043.1645
    • Windows Feature Experience Pack: 120.2212.4170.0
  • WSL 2 with Ubuntu-20.04
  • Docker Desktop 4.7.1 (77678) with WSL 2 based engine

Update 1

As suggested by @Martin, I tried downloading https://www.amazontrust.com/repository/AmazonRootCA1.pem, put it inside /tmp in WSL Ubuntu, and reran the command curl --cacert /tmp/AmazonRootCA1.pem -vfsSL https://apt.releases.hashicorp.com/gpg to no avail:

curl --cacert /tmp/AmazonRootCA1.pem -vfsSL https://apt.releases.hashicorp.com/gpg
*   Trying 52.222.214.72:443...
* TCP_NODELAY set
* Connected to apt.releases.hashicorp.com (52.222.214.72) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /tmp/AmazonRootCA1.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
casparjespersen
  • 113
  • 1
  • 7
  • This is most likely a problem with your local root-ca file... every OS must have a list of root CAs which are trusted by your OS, or in your case, the docker subsystem. If the root CA of the remote host is not on your local list of trusted CAs, the connection is rejected. Is the file ```/etc/ssl/certs/ca-certificates.crt``` present? – Martin May 09 '22 at 06:42
  • Yes it is. It contains numerous certificates, it appears. – casparjespersen May 09 '22 at 10:42
  • download ```https://www.amazontrust.com/repository/AmazonRootCA1.pem```, put it inside ```/tmp```, and rerun the command like this: ```curl --cacert /tmp/AmazonRootCA1.pem -vfsSL https://apt.releases.hashicorp.com/gpg``` - if this succeeds, your ca-certificates bundle is missing the amazon root ca ! – Martin May 09 '22 at 10:59
  • @Martin I just tried, and unfortunately this does not fix the issue. I have updated my original question with verbose output details from the rerun. – casparjespersen May 09 '22 at 11:07
  • Is it able to connect to the crl? – Greg Askew May 09 '22 at 11:19
  • @GregAskew I'm a novice in this area. I'm going to need a bit more direct pointer :) connect to crl .. What is that, how do I test it? – casparjespersen May 09 '22 at 11:21
  • The endpoint needs to be able to validate the certificate. One validation is connecting to the certificate revocation list crl on tcp/80. – Greg Askew May 09 '22 at 11:22
  • Sorry but that did not help me on how I can check whether it's possible. Can you provide a link to a guide? – casparjespersen May 09 '22 at 11:37
  • GregAskew is referring to a link which is written directly inside the certificate. (open the link in a browser and click on "view certificate") I think it is this link ```http://crl.sca1b.amazontrust.com/sca1b-1.crl``` - try to access this link from within the container... – Martin May 09 '22 at 11:52
  • Ok. Looking on my Windows OS I see the certificate CRL Endpoint being http://gateway.zscaler.net/zscaler-zscrl--4.crl -- and I am able to retrieve this using CURL within the WSL. – casparjespersen May 09 '22 at 11:57
  • Is it possible that you have a gateway / firewall in between your docker container and the internet which intercepts SSL connections ? This CRL you have posted there does not look like it comes from the original certificate... – Martin May 09 '22 at 11:58
  • That is certainly quite possible. Our IT department controls this remotely. – casparjespersen May 09 '22 at 12:07
  • please execute the following and list the trust chain inside your answer please: ```echo "" | openssl s_client -connect apt.releases.hashicorp.com:443 -servername apt.releases.hashicorp.com:443 | openssl x509 -text -noout```. – Martin May 09 '22 at 12:09
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/136157/discussion-between-casparjespersen-and-martin). – casparjespersen May 09 '22 at 12:14

1 Answers1

4

We have finally figured it out. The issue here was that the IT department installed a firewall which intercepts SSL connections (many firewalls do this to be able to block "bad" traffic which would go through undetected because of the encryption).

To be able to intercept a SSL connection, The firewall puts its own certificate on the link, and acts as a proxy. That is where the issue is coming from: The docker container does not trust the CA used by your firewall, and therefore rejects establishing the connection. Obviously, for your windows machine the CA has been added to the trust store already - this needs to be done for your docker container as well - or you need to add the --insecure switch to your curl command to ignore those certificates error.

Note that curl correctly throws an error, since the https - certificate mechanism was invented to be able to detect a "Man in the Middle" attack - and this is exactly what is happens here. (I will just skip my opinion here about firewalls breaking SSL security to be able to inspect packet contents...)

To fix this, you need the CA file from your gateway which "signs" all those certificates, and add this CA to the trusted list.

Just for completeness, this is the certificate you see:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6a:7b:6c:fd:70:b2:68:1a:56:e3:11:3c:43:ed:ae:29
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = US, ST = California, L = Sunnyvale, O = Bad Server Certificate [invalid server certificate], CN = Bad Server Certificate
        Validity
            Not Before: Jan 26 07:08:23 2011 GMT
            Not After : Nov 11 07:08:23 2284 GMT
        Subject: CN = apt.releases.hashicorp.com
[cut]

this is the original one:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:57:ef:0e:70:b2:68:1a:56:e3:11:3c:43:ed:ae:29
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
        Validity
            Not Before: Apr 27 00:00:00 2022 GMT
            Not After : May 26 23:59:59 2023 GMT
        Subject: CN = apt.releases.hashicorp.com
Martin
  • 1,869
  • 6
  • 16
  • 1
    Thank you. It's not feasible/possible for me to add an --insecure flag to the curl command. So I'm interested in the proper fix. I went to the certificate store on Windows and exported the CER file from the Zscaler Root CA certificate, and now it works when doing `curl --cacert ca.cer -fsSL https://apt.releases.hashicorp.com/gpg` Hooray! Finally, what is the proper way of adding this certificate "globally", so I don't have to specify it every time? – casparjespersen May 09 '22 at 13:20
  • 2
    Nevermind - found on: https://ubuntu.com/server/docs/security-trust-store. Thanks a ton! – casparjespersen May 09 '22 at 13:27
  • Glad I was able to help! – Martin May 09 '22 at 14:29