"Update Your Amazon RDS SSL/TLS Certificates by October 31, 2019"

Question

I, like a lot of people, received an email saying to update my RDS instance to use the new rds-ca-2019 certificate for SSL connections (previous being rds-ca-2015 which expires March 5, 2020). Their documentation about the process is a little sparse and says things like "Update your database applications to use the new SSL/TLS certificate." and "Import the certificate into your operating system." with no further details on changes required on the client side.

When I initially set things up, I didn't install any certificates and used a vanilla Ubuntu 18.04 EC2 image. The RDS instance was set to use rds-ca-2015 and when I connected to RDS with psql it reported that it was properly using TLSv1.2. If I look at root certificates installed in the OS I find 4 "Amazon Root CA" certs numbered 1 through 4. Those don't expire until 2038 and 2040.

So, my question has 2 parts:

How did the SSL/TLS properly work initially if I had never installed the RDS certs and intermediary certs provided by Amazon?
If I've changed the RDS database instance to use rds-ca-2019 and it seems to "just work" is there anything more I need to do?

My best guess is that the intermediate certificates are provided by the server when a TLS connection is established. — Tim Tisdall, Oct 21 '19 at 14:31

Tim Tisdall · Accepted Answer · 2019-10-28T14:08:38.283

14

The default sslmode for PostgreSQL is prefer which means it will encrypt the connection with the certificate provided by the server but will not verify it. If I were to change the sslmode setting to verify-ca or verify-full then I would need to install the intermediate certs in a particular directory and then it would do proper verification.

As I'm not concerned about a MITM attack on my VPC, I don't think I'll bother switching to the 'verify' modes.

edited Oct 28 '19 at 14:08

answered Oct 22 '19 at 16:12

Tim Tisdall

623
1
5
17

If only SOC2 auditors shared your sentiment! :) – contactmatt Jan 13 '21 at 18:23

score 8 · Answer 2 · answered Oct 22 '19 at 13:14

The RDS certificate in question is an intermediate certificate. You might also know it as a CA certificate. When I use MySQL Workbench, for instance, I have to specify that

I want to use SSL(TLS)
Use the RDS CA chain file to verify the certificate

How did the SSL/TLS properly work initially if I had never installed the [certificate]?

Depends how your system is set up. CA certificates simply provide a trusted authority for the presented certificate. It's quite possible to set up something that will accept any certificate at all, without attempting to verify it (i.e. you use a self-signed certificate). Another option is there is something already in your CA store that trusts it implicitly. This is less likely, but not impossible.

If you're doing this locally (such as you have an EC2 instance in the same VPC as your RDS instance) you might not even need SSL.

If I've changed the RDS database instance to use rds-ca-2019 and it seems to "just work" is there anything more I need to do?

No. It's confusing, but if you're connecting and not getting any certificate errors, I wouldn't worry about it.

I hadn't thought about the possibility that it was using the cert without verifying (as I assumed that the default settings would be to reject certs not signed by a CA installed in the OS). It seems with `psql` it looks for certs in a certain directory and doesn't use the OS's CA certs. The default setting for `sslmode` is `prefer` which doesn't do verification but uses the server's provided cert if one exists. — Tim Tisdall, Oct 22 '19 at 16:06

vinh · Answer 3 · 2020-01-21T15:44:33.220

In ubuntu, add the ca-cert as described here: https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate

wget https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
sudo mkdir /usr/local/share/ca-certificates/aws
sudo mv rds-ca-2019-root.pem /usr/local/share/ca-certificates/aws
sudo openssl x509 \ 
    -in /usr/local/share/ca-certificates/aws/rds-ca-2019-root.pem \
    -inform PEM \
    -out /usr/local/share/ca-certificates/aws/rds-ca-2019-root.crt
sudo update-ca-certificates

: sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Adding debian:rds-ca-2019-root.pem
done.
done.

This link describes how to set the ssl cert location for a django application: https://www.digitalocean.com/community/questions/how-to-connect-managed-database-postgres-with-ssl-mode-varify-full-in-django-app

DATABASES = {
'default': {
    'ENGINE': 'django.db.backends.postgresql_psycopg2',
    'NAME': '<name>',
    'USER': '<user>',
    'PASSWORD': '<password>',
    'HOST' : '<host>',
    'PORT' : '25060',   
    'OPTIONS':{
        'sslmode':'verify-full',
        'sslrootcert': os.path.join(BASE_DIR, 'ca-certificate.crt')

}

This post in Stackoverflow https://stackoverflow.com/a/58214922/1415254 describes how to connect using commandline parameters for psql.

psql "host={hostname} sslmode=prefer sslrootcert={ca-cert.pem} \
sslcert={client-cert.pem} sslkey={client-key.pem} port={port} user={user} \
dbname={db}"

Also

psqlrc and ~/.psqlrc

Unless it is passed an -X or -c option, psql attempts to read and execute commands
from the system-wide startup file (psqlrc) and then the user's personal startup
file (~/.psqlrc), after connecting to the database but before accepting normal
commands. These files can be used to set up the client and/or the server to taste,
typically with \set and SET commands.

And more detail here (right at the end): https://info.crunchydata.com/blog/ssl-certificate-authentication-postgresql-docker-containers

# the first parameter specifies which TLS mode to use to connect
export PGSSLMODE="verify-full"
# the following two parameters point to the client key/certificate
export PGSSLCERT="`pwd`/certs/client.crt"
export PGSSLKEY="`pwd`/keys/client.key"
# this parameter points to the trusted root CA certificate
export PGSSLROOTCERT="`pwd`/certs/ca.crt"

Full list of environment variables here: https://www.postgresql.org/docs/9.2/libpq-envars.html

score 4 · Answer 4 · answered Feb 02 '20 at 08:46

Just like the post OP answered himself, postgres has the default sslmode set to prefer, and this is the excerpt from the doc:

I don't care about encryption, but I wish to pay the overhead of encryption if the server supports it.

So by default, the pg driver will not verify the certificates unless specified; and this is exactly why to OP's original questions, it works out of the box at the beginning, and also works after the RDS is upgraded to rds-ca-2019.

One of the environment variables for connecting to postgres is via DATABASE_URL, in the form of

postgres://username:password@host/database?sslmode=verify-full&sslrootcert=config/ca/rds-combined-ca-bundle.pem

and this is where you can specify the sslmode and sslrootcert if you decide to verify the intermediate certs. The content of sslrootcert should be one of the following...

https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem

HTH

score 3 · Answer 5 · answered Feb 13 '20 at 03:04

There are two sides to this certificate update:

On the server side: Any RDS instances created before 2019-11-01 require a restart. This incurs an outage of ~5 minutes. This will update them to have the new certificates in place.
If your clients use SSL (off/prefer by default), then you need to update the certificates (intermediate+chain) from AWS on wherever your clients run.

Postgres uses "prefer" as the default way for clients to connect, which means they will try SSL if available, but fall back if not. So existing clients with default connection configuration will continue to operate.

score 2 · Answer 6 · answered Jan 10 '20 at 06:02

Before upgrading Certificate authority in RDS to rds-ca-2019, without connection interruption, you can upgrade certificate on client-side.

If your RDS has rds-ca-2015, you should upgrade the client-side key with this https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem.

As per AWS document https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html It says rds-combined-ca-bundle.pem file has both certs intermediate and root.

Once your applications have combined-ca file then you should proceed to upgrade your RDS to Certificate authority rds-ca-2019.

This way without the downtime you can upgrade Certificate authority in RDS to rds-ca-2019.

score 1 · Answer 7 · answered Oct 22 '19 at 10:55

Any new RDS DB instances created after November 1, 2019, use the new certificates by default. If your RDS instance is created before the above-mentioned date, You need to update the certificate.

To change the Certificate Authority from rds-ca-2015 to rds-ca-2019 for a DB instance

Sign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/.
In the navigation pane, choose Databases, and then choose the DB instance that you want to modify.
Choose Modify. The Modify DB Instance page appears. refer attachments
In the Network & Security section, choose rds-ca-2019. refer attachments.
Choose Continue and check the summary of modifications.
To apply the changes immediately, choose "Apply immediately". Choosing this option causes an outage.
On the confirmation page, review your changes. If they are correct, choose Modify DB Instance to save your changes.

My question is about the client side and you're explaining how to update the server side. — Tim Tisdall, Oct 22 '19 at 12:27
the emails from amazon and the links it has and all of that just leads one to believe that they need to install something on the server they are connecting from. edit: so thats wrong? just change it from the admin tool? — Lassi Kinnunen, Dec 01 '19 at 09:39

"Update Your Amazon RDS SSL/TLS Certificates by October 31, 2019"

7 Answers7