20

If I try to access our HTTPS server that has certbot-issued certificate from debian 9, I get the following error:

 # curl -v https://hu.dbpedia.org/
 *   Trying 195.111.2.82...
 * TCP_NODELAY set
 * Connected to hu.dbpedia.org (195.111.2.82) port 443      (#0)
 * ALPN, offering h2
 * ALPN, offering http/1.1
 * Cipher selection:      ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
 * successfully set certificate verify locations:
 *   CAfile: /etc/ssl/certs/ca-certificates.crt
   CApath: /etc/ssl/certs
 * TLSv1.2 (OUT), TLS header, Certificate Status (22):
 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
 * TLSv1.2 (IN), TLS handshake, Server hello (2):
 * TLSv1.2 (IN), TLS handshake, Certificate (11):
 * TLSv1.2 (OUT), TLS alert, Server hello (2):
 * SSL certificate problem: certificate has expired
 * Curl_http_done: called premature == 1
 * stopped the pause stream!
 * Closing connection 0
 curl: (60) SSL certificate problem: certificate has expired

However, if I try the same command from debian 10, it succeeds.

I tried to simply copy all ca-certificates from a debian 10 VM to the debian 9 VM (to /usr/local/share/ca-certificates) with rsync, and then ran update-ca-certificates which seemingly added 400+ certificates. Unfortunately, it did not help. This is no wonder, as it seems that there are the same certificates on both debian 9 and 10, apparently.

My question is: How can I access sites with certbot certificates from debian 9 machines without ignoring certificate verification altogether

P.Péter
  • 499
  • 2
  • 6
  • 17
  • @GeraldSchneider Unfortunately, that is not an option, at least in the short term. I would like to have a temporary solution until the upgrade can be made. – P.Péter Sep 30 '21 at 16:56
  • Fun fact: old (~4 year old) versions of OSX suffer from the same problem. They also probably use and old openssl version. – P.Péter Oct 06 '21 at 08:18

6 Answers6

26

First off, Debian 9 is EOL. But as the clients may not be under your control, you may of course want to try to cater to them in this breakage.

I assume that while the question only mentions certbot, that it's really specifically about Letsencrypt.
(The tool certbot itself is an ACME protocol client which is also used with other ACME-based CAs, so there is some room for confusion here.)

The problem at hand would appear to be the combination of:

  • The old Letsencrypt root ("DST Root CA X3") has expired
  • The new default LE chain tries to be "extra compatible" by presenting an optional extension of the chain where the new root ("ISRG Root X1") is presented as a cross-signed intermediate for the old root (as very old Android versions still accept the expired root, but do not have the new root)
  • Openssl 1.0 has a bug causing it to just try the first chain it sees and if it doesn't like that, it doesn't look at any other possibilities (ie, the shorter new chain ending at X1 vs the longer "compatibility" extension of that chain going through X1 to X3).
  • libcurl3 on Debian 9 is linked to libssl 1.0

If you instead present the new LE certificate chain that does not try to be extra compatible, just ending at the new root (X1), it allows libssl 1.0 to work (but then you lose compatibility with really old Android instead).

Other than that, other CAs (ACME or otherwise) is probably the option to consider.

Håkan Lindqvist
  • 33,741
  • 5
  • 65
  • 90
  • 1
    This was the best explanation for the problem, thank you! – P.Péter Oct 01 '21 at 06:43
  • 4
    By the way, the most "debian" way of disabling a certificate seems to be running `dpkg-reconfigure ca-certificates` and deselecting the offending one from the list. – P.Péter Oct 01 '21 at 06:49
  • 3
    Debian 9 is not really End-Of-Life, Debian 8 reached EOL this summer but Debian 9 continues to be supported with security updates. – Orphans Oct 04 '21 at 11:44
  • It is a great explanation of the cause of the problem, but is it possible to fix it without upgrading to Debian 10 ? – bN_ Oct 26 '21 at 11:47
  • 2
    Edit: @P.Péter using `dpkg-reconfigure ca-certificates` to deselect "DST Root CA X3" solves the problem. The accepted answer actually provides explanations but no solution. It should be updated with @P.Péter comment. – bN_ Oct 26 '21 at 11:56
  • I haven't tried latest version of openssl yet , but I hit the same issue on `openssl 1.1.1c` in my Debian system , and solve the issue by removing expired DST X3 cert and adding new ISRG X1 cert (by running `dpkg-reconfigure`) . Check out [this article](https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/) for detail – Ham Nov 30 '21 at 04:47
26

Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.

Below solution works on Debian Jessy 8 and should on Stretch 9 missing updates too. I just tested it on docker run -it debian:jessie bash, apt-get update && apt-get install curl.

Before:

# curl -I https://hu.dbpedia.org
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html
....

First check if you have offending DST Root CA X3 cert present:

# grep X3 /etc/ca-certificates.conf 
mozilla/DST_Root_CA_X3.crt

Old Debian releases have the proper ISRG Root X1 present too:

# grep X1 /etc/ca-certificates.conf 
mozilla/ISRG_Root_X1.crt

This is going to disable X3:

# sed -i '/^mozilla\/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f

Your domain responds to curl fine:

# curl -I https://hu.dbpedia.org/
HTTP/1.1 200 OK
...

Again, plan an upgrade please.

gertas
  • 1,007
  • 10
  • 11
  • 1
    This removing/commenting out the cert in the conf file worked for us as well. Thank you for the help. – Ray Oct 01 '21 at 16:29
  • 1
    I can confirm that this works on Debian Stretch too. – Dominik Oct 07 '21 at 07:36
  • Works on three of my Debian GNU/Linux 10 (buster) servers. Four of them went down in 1 day and I had to strip them mostly of curl for them to work. One website hosted on Debian Buster isn't working still, but it may be a different error I suppose as the previous three websites went down on October 1, but this one with the remaining error went down January 4ish. Thank you! – luminol Jan 13 '22 at 20:36
5

DST Root CA X3 root certificate expired on Sep 30 14:01:15 2021 GMT. It was used as one of certification paths for Let’s Encrypt certificates Older cURL version has a bug that will cause expired root to fail connection instead of trying other roots in local ca store.

As of 30 september 2021 ca store for curl (https://curl.se/docs/caextract.html) still contains expired DST Root CA X3 root certificate so updating it won't fix problem. You can either update your cURL (which might be quite challenging in some situations) or edit local ca store (f.e. /etc/pki/tls/certs/ca-bundle.crt) and manually remove certificate after line "DST Root CA X3"

ndd
  • 139
  • 5
2

I've just had this problem with a Docker image based on Debian Stretch. The solution has been easy, I suppose Debian has patched libssl1.0.2 to fix the chain selection issue Håkan has mentioned. In my case, updating just the certificates and libssl1.0.2 by running apt install -y libssl1.0.2 ca-certificates (with apt update for containers beforehand) has been enough.

Nikolai Prokoschenko
  • 553
  • 1
  • 7
  • 21
2

For Debian 8 and 9, I just do that for updating cert on host and no longer have the error 60: SSL certificate problem: certificate has expired : https://github.com/xenetis/letsencrypt-expiration

Just run, it should work :

wget -O - https://raw.githubusercontent.com/xenetis/letsencrypt-expiration/main/letsencrypt-expiration.sh | bash
Nicolo
  • 31
  • 4
1

Following Nicolo's answer1, I ran the script on my aging Debian 9 server and still was getting the same certificate expired error. update-ca-certificates man page says

Furthermore all certificates with a .crt extension found below /usr/local/share/ca-certificates are also included as implicitly trusted.

The script he shared, as of today, download ISRG & Let’s Encrypt certificates as .pem files, so update-ca-certificates doesn’t automatically include them. After making a soft link to these .pem files, but with a .crt extension, running update-ca-certificates again does indeed include them and the expired certificate error goes away. Thank you for your answer!

mk05
  • 11
  • 3
  • I always have a script that copies the PEM files into the daemon’s config directory instead of a symlink. – John Greene Apr 24 '22 at 19:08
  • I've updated the script here https://github.com/fayak/letsencrypt-expiration/blob/main/letsencrypt-expiration.sh – zarak Jul 06 '22 at 14:29