0

The ERR_CERT_DATE_INVALID error, I'm sure we're all familiar with, is below

enter image description here

Visiting the same site from numerous other locations, web clients, etc shows a valid certificate.

It's issued by let's encrypt (cert-bot) and auto-renews. Thousands of people have accessed the web app without issue.

Is this a possible MITM attack? Is it possible to cache an old cert before renewal? I've never seen this without the certificate actually being out of date... What can cause this if the certificate is confirmed valid/not expired?

Here's the cert info from accessing the site anyway I can: site cert One intermediary:
enter image description here
and root:
enter image description here

However, the Root expiration date is different when reviewing on SSL labs: ssl labs expiration date

And here is further output from SSL Labs around the expired cert causing the issue: ssl labs further detail

TCooper
  • 336
  • 1
  • 8
  • 1
    Hard to tell without further context. Could be MITM, could be local configuration problem ... whatever. Please have a look at the detailed message in Advanced and also have a look at the certificate and compare it to what you get on other machines. Then update your question with these information. – Steffen Ullrich Dec 21 '21 at 18:00
  • Thanks, will reach out and add here when I get responses – TCooper Dec 21 '21 at 18:01
  • 3
    Looking deeper into the blurry image - there seems to be something about ERR_CERT_DATE_INVALID. Based on this the error might be caused on your system having the wrong time. – Steffen Ullrich Dec 21 '21 at 18:03
  • Or possibly an expired certificate somewhere else in the chain, that is not cross-signed with a certificate that chains up to another root certificate in your trust store. – mti2935 Dec 21 '21 at 18:09
  • @SteffenUllrich Good eye. And sorry for image quality, just added blue boxes to what I got... Trying to get the cert details etc, now. The system time makes sense, but they'd have to be wayyyy off. This cert is valid Nov. 8 - Feb 6, pretty much dead middle of it, and the screenshot is from last night. Will check though – TCooper Dec 21 '21 at 18:12
  • In that case make sure the right LE intermediate certificate is included in the certificate chain. Their old one expired a number of weeks ago. Sending a valid certificate with an expired intermediate gives this error too. Testing the chain on a site like SSLlabs should show this. – Teun Vink Dec 21 '21 at 18:22
  • @TeunVink Thanks, I've only ever seen it show up correctly/as expected. Waiting on details from user that is getting this error - but will be sure to check that as well. Kind of seems like NEI until we can see the details from the cert. – TCooper Dec 21 '21 at 18:25
  • I’ve seen different browsers behave differently on expired intermediate certificates. – Teun Vink Dec 21 '21 at 18:26
  • @mti2935 Can you explain a bit further, I want to make sure I understand you correctly; why would a user have a different trust chain? – TCooper Dec 21 '21 at 18:27
  • @TeunVink One nice thing about this app is I force users to Chrome, or simply load a message that says "This only works on Chrome" in any other browser (yes, users that spoof their user agent can get by of course... but that's not this user). – TCooper Dec 21 '21 at 18:28
  • 1
    @TCooper Even if the leaf certificate for your site is still valid, your browser may show an expired certificate error if one of the other certificates in the chain is expired. The problem that CA's face is that older certificates tend to be widely distributed, but expire sooner; while newer certificates expire later but are not as well distributed. So, to get the best of both worlds, CA's will sometimes 'cross-sign' certificates. See https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020 for more info. – mti2935 Dec 21 '21 at 19:13
  • 1
    Have you played around with the clock/time settings on that client? – schroeder Dec 21 '21 at 19:23
  • 2
    "but they'd have to be wayyyy off" -- no they don't actually. A small change can cause a time-based error. – schroeder Dec 21 '21 at 19:25
  • @schroeder I don't have access to the client getting the error, reported to me remotely. What do you mean about a small change? I thought they would have to be outside the valid date of the cert, but it can cause an error just by not matching what the server(or..? browser reads from os right?) expects? – TCooper Dec 21 '21 at 20:49
  • https://serverfault.com/questions/852419/how-much-clock-asynchronicity-can-secure-protocols-tolerate – schroeder Dec 21 '21 at 20:54
  • There's a trick to force the Windows system time to mimic a bad clock and trigger the error, but I can't remember it. Get the client to sync their time with a time server and test again. – schroeder Dec 21 '21 at 20:57
  • 2
    @TCooper The root certificate (and possibly the intermediate cert) that you show in your chain has been cross-signed. See https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/. It's possible that the system that is showing the cert expiry error is doing so, for the reason explained in the link above - specifically, this system may have the older DST Root CA X3 cert installed (which expired Sep 30, 2021), but not the newer ISRG Root X1 cert. – mti2935 Dec 21 '21 at 21:35
  • @mti2935 What a mess. I've been lucky enough to never run into this before (and lucky enough I've been almost entirely in the let's encrypt era of free, easy, auto-renewing certs, in general). Learning a lot about these patchwork fixes today. Thanks for all the info! When I confirm what the issue is/was I'll post here so someone can answer for real... Haven't heard anything back yet though – TCooper Dec 21 '21 at 22:46
  • @TeunVink just actually visited SSL labs... great resource(far superior to 'why no padlock?'), thanks again – TCooper Dec 21 '21 at 22:54
  • @TCooper When you scan your site using SSL Labs, do the expiration dates of all the certificates in the chain served by your server match those in the images that you posted? – mti2935 Dec 22 '21 at 11:00
  • @mti2935 nope, you're spot on. I took a screenshot, adding to the question. The root is the same name, ISRG root X1, but the expiration is in 2024, and I can expand to see the expired Sep 30, 2021 expired cert you mentioned. Going to take steps in link you posted in comment, but if you want to make an answer... – TCooper Dec 22 '21 at 16:51
  • 1
    @TCooper, I figured that might be the case. It sounds like your server is serving the DST Root CA X3 cert that expired Sep 30, 2021. Most browsers have the newer (unexpired) ISRG Root X1 installed, and will use this root cert instead of the DST to build a valid trust chain, as you show in the images that you posted. But, some devices may not do this, and in this case will show the expired cert error that your client reported to you. You might want to replace the DST Root CA X3 cert with the ISRG Root X1 cert in the cert bundle that your server serves. – mti2935 Dec 22 '21 at 17:17
  • @mti2935 I'm a little at a loss here though, the server does serve the ISRG Root X1, it's just cross signed with the older CA X3(?). But following the steps from the link you provided does not change ssl labs output. When ssl labs says its "In trust store" isn't that ssl labs trying to mimic a browser, not what the server serves?(added more ssl labs details in question) - also wouldn't it break in more than one client, if the server was still serving the old cert? I'm thinking this is local to the machine / they have the old cert cached somehow? – TCooper Dec 22 '21 at 19:05
  • @TCooper Thanks for posting the output from SSLLabs. This is helpful. Do you see that the cert chain that your server is serving chains up to ISRG Root X1, and that this cert is signed by DST Root CA X3? The DST Root CA X3 is the one that is expired. – mti2935 Dec 22 '21 at 19:40
  • @TCooper Also in the output from SSLLabs, you can see what Mozilla does to build a valid trust chain - it replaces the ISRG Root X1 cert (which is signed by the DST cert) with a different ISRG Root X1 which is self signed, and this ISRG Root cert is in Mozilla's trust store. Try changing the cert chain that your server serves so that it is the same as what Mozilla is doing, and I think that will solve the problem. – mti2935 Dec 22 '21 at 19:40
  • @mti2935 do you happen to know where let's encrypt reads the chain file from? I'm hitting google now, but if I can learn from those more experienced, always want to. How common is naming a root cert the same thing, with it being a different cert? A lot of my confusion stemmed from thinking ISRG Root X1 was the same cert in both trust paths, but the fingerprint makes it apparent this isn't the case – TCooper Dec 22 '21 at 20:51
  • @TCooper, I'm happy to help if I can. I'm not sure what you mean by 'Let's Encrypt reading the cert chain from'. LE does not read the cert chain, they simply sign your leaf certificate. Then, your server serves the entire cert chain (the leaf cert, the intermediate cert(s), and the root cert) to the browser. WRT the ISRG certs - the expired one is not actually a root cert. A root certificate must be self-signed. – mti2935 Dec 22 '21 at 21:12
  • @TCooper, Notice that the expired ISRG cert is not self-signed, (it is signed by the DST cert, which is self-signed), whereas the one that expires in 2035 is self signed. You can download the self-signed ISRG cert from https://letsencrypt.org/certs/isrgrootx1.pem. If your server serves a chain that chains up to this cert, that should solve the problem (assuming browsers have this cert in their trust store, which they should, being that this is a very widely used root cert). – mti2935 Dec 22 '21 at 21:13
  • @TCooper, and in that case, your cert chain will be like: `leaf cert -> R3 (exp 9/2025) -> ISRG Root X1 (exp 6/2035)` which is the same as security.stackexchange.com and many other sites on the web. So, if your client's browser does not give a certificate error for security.stackexchange.com, then this chain should work for your site as well. – mti2935 Dec 23 '21 at 14:07

1 Answers1

1

After some back and forth with OP in the comments, it turns out that the source of the problem is that OP’s web server is serving a certificate chain containing one or more expired certificates. The reason that the problem only manifests itself on rare occasions is that OP’s leaf certificate is issued by Let’s Encrypt (LE), and LE cross-signs its intermediate and root certificates. Most modern browsers are able to create a valid certificate chain to a trusted root certificate using the cross-signed certificates.

It took me a while to get my head around the concept of cross-signing, and how modern browsers are able to use cross-signed certificates to compensate for expired certificates in the chain served by the server - so I thought it would be constructive to summarize the comments into an answer, and add some explanation as to how all of this works under the hood.

The certificate chain that OP’s site serves is:

opsite.com → R3 (exp 9/29/2021) → DST Root CA X3 (exp 9/30/2021)

Notice that the intermediate certificate (R3) and the root certificate (DST DST Root CA X3) are both expired. LE started warning its users about the expiring root certificate well in advance, wherein it reads, ‘What should you do? For most people, nothing at all!’.

Indeed, rather than failing to connect to OP’s site because of the expired certificates, most browsers will build this chain instead (as OP shows in his question):

opsite.com → R3 (9/15/2025) → ISRG Root X1 (exp 6/4/2035)

So, the first takeaway from this is that the trust chain that the browser builds (and shows in the TLS connection information) is not necessarily the same as the one that the site serves.

But more importantly, how are browsers able to compensate for the expired certificates in the certificate chain that the site serves, with the site admin doing ‘nothing at all’ - and build this valid trust chain of unexpired certificates? This where cross-signing comes into play.

LE has two different intermediate certificates named R3. Let’s call them R3 version 1 and R3 version 2:

enter image description here

First, notice what is the same about the two certificates: They both have the same subject name, and they both have the same public key. Now, notice the differences: R3 version 1 is signed by DST Root CA X3, whereas R3 version 2 is signed by ISRG Root X1; and R3 version 1 is expired whereas R3 version 2 does not expire until 2025. Of course, the fingerprints are different as well, because they have different issuers and different expirations. (LE refers to R3 version 1 as the ‘cross-signed certificate’).

So, this is what modern browsers do when they connect to OP’s site:

  1. Download the certificate chain from OP’s site.
  2. The leaf certificate is not expired, and the issuer is R3, so look for a certificate in the certificate chain with the subject ‘R3’.
  3. Found one in the certificate chain from the site (R3 version 1). But, it’s expired, so we can’t use it. Is there another certificate with the subject ‘R3’ somewhere that I can use? Lo and behold, there is! (R3 version 2). And it doesn’t expire until 2025, so let’s see if we can use it.
  4. Is the signature on the leaf certificate by R3 version 2 is valid? Well, if the signature on the leaf certificate by R3 version 1 was valid, then the signature on the leaf certificate by R3 version 2 will be valid as well, because R3 version 1 and R3 version 2 both have the same public key.
  5. Continue validating the chain until we come to a trusted root certificate (in this case, ISRG Root X1).

So, the certificate chain built by the browser (using R3 version 2 instead of R3 version 1) is completely kosher.

But, wait a second… Back up to step 3. Where did R3 version 2 come from? Modern browsers maintain a cache of intermediate certificates. If R3 version 2 is stored in the cache, the browser will find it there. X.509 also has the AIA extension, which is used to specify a URL from where the issuing certificate can be downloaded. Being that OP’s leaf certificate was issued by LE, this field was populated with a URL to a server in LE’s domain where the intermediate certificate can be downloaded, and LE now has R3 version 2 at this URL.

So, why didn’t it work in the case that OP mentioned in his question? It’s hard to say without more information, but there are a number of possibilities. It could have been that this was an older browser that doesn’t perform the steps above to handle cross-signed certificates. Or, it could have been that for whatever reason, this browser was not able to find R3 version 2. Or, perhaps this browser is behind on updates, and does not have ISRG Root X1 in its trust store.

Final note: This shows how cross-signing can be used to compensate for expired certificates in the certificate chain served by a site. But, cross-signing serves a more fundamental purpose – it allows for a browser to create multiple certificate chains to trusted roots. Consider a case where a browser connected to OP’s site before Sep 2021. There would have been two chains to a trusted root that the browser could have built – one to DST Root CA X3 and another to ISRG Root X1. In other words, the browser would only have needed either one of those root certificates installed, in order to build a valid trust chain. So, cross-signing provides CA’s with a way to phase-in new root certificates. The certificate chain served by the server can chain-up to the CA’s new root certificate, but if a client does not yet have the new root certificate installed, it can still chain to an older (more widely distributed) one, by way of a cross-signed certificate.

mti2935
  • 19,868
  • 2
  • 45
  • 64
  • Great and thorough answer! I had forgotten about this question, but hopefully I can come back to clean it up sometime in the next week. Thanks for working through all the details – TCooper Jan 10 '22 at 01:53