OpenSSL fails to detect expired intermediate CA certificate in s_client SSL connection test

Question

By accident, I have an expired intermediate certificate at the end of my chain file in my Dovecot server's SSL configuration. It's enough of a problem that my Android e-mail client refuses to use it, although Apple Mail lets it go (??!). Indeed, the expiration just happened hours ago. openssl x509 -in ... shows:

    Serial Number:
        13:ea:28:70:5b:f4:ec:ed:0c:36:63:09:80:61:43:36
    Signature Algorithm: sha384WithRSAEncryption
    Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
    Validity
        Not Before: May 30 10:48:38 2000 GMT
        Not After : May 30 10:48:38 2020 GMT
    Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority

But this command:

openssl s_client -showcerts -verify_return_error -connect imap.example.com:993

fails to flag the problem (while outputting the expired certificate!). The OpenSSL package version is: 1.1.1g-1+ubuntu18.04.1+d

CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = imap.example.com
verify return:1

How do I create an OpenSSL verification test to find and flag this? I have searched online already quite a bit and found nothing to address expiration down a few rungs in a public chain. The closest question is: Why is my SSL certificate untrusted on Android? but this only deals with a missing link in a 4-certificate chain. My guess as to why Apple Mail accepts the error is that MacOS has cached its own non-expired version of the same intermediate CA.

EDIT

On the server, the following:

/usr/share/ca-certificates/mozilla/USERTrust_RSA_Certification_Authority.crt

is now self-signed, so openssl must be silently substituting this one (edit: tested by hiding this cert; the expiration is now detected!) but my goal is to be sensitive enough to detect the Android's complaint. I am on Android 10 but one (May 4) update behind the latest.

Answer 1

Credit to SSLMate for the great explanation.

Basically the AddTrust External CA Root certificate is now surplus to requirements, and as of May 30th 2020, that's the certificate that just expired. There are likely other certificates in your CA bundle which could be used to verify the main certificate you have, but older software (OpenSSL 1.0 and the like) is failing because it's choosing the certificate chain that has just expired, rather than the one that would work. The solution is to remove the expired certificate from your certificate file, but you probably already knew that.

However, knowing what's vulnerable is the key to writing the test you need - you need to wrap an old vulnerable OpenSSL version in something that can be exercised as part of your test. I've been using curl on MacOS because that refuses to validate a cert with the AddTrust certificate in the chain, but we've seen command line openssl detect it as well, not sure exactly what version we're running.

Of course, if you know it's specifically this certificate that's bad, you can always just find it by its signature:

openssl x509 -noout -fingerprint -in ../AddTrust\ External\ CA\ Root-2020.crt`

and compare the response to

SHA1 Fingerprint=02:FA:F3:E2:91:43:54:68:60:78:57:69:4D:F5:E4:5B:68:85:18:68

which is this particular bad certificate.

I'm hoping this particular expired cert vulnerability is a rarity, but if you want to make a more general test, you want to look for the case where more than one possible chain exists, and one is valid but another is not.