How to automate certbot certificate renewal on Ubuntu 20.04

Question

I'm running certbot on Ubuntu 20.04 in AWS, installed as a snap package. I'm not sure if certbot renewal is running properly. I'd appreciate some help working out how to best get it working.

This is a new server, which I turn on and off while I'm getting it ready for production. It runs about 8 - 10 hours a day at the moment. It's not often running at midnight, which I think is when the cron job runs. It will be on 24/7 in a few days once I finish the configuration.

One thing I found is this question answer saying

You shouldn't have to set up anything. Any recent Debian/Ubuntu install of certbot should install a systemd timer and a cron job (and the cron job will only run certbot if systemd is not active, so you don't get both running).

It looks to me like the certbot timer isn't running, and if it did it appears to be pointing at /dev/null. Because systemd is active I wonder if the cron job is running.

Timers and systemd

I found a comment there may be an issue with timers and snap so maybe this is a known issue.

systemctl list-timers

The timer doesn't seem to run

NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES
Wed 2021-03-17 23:44:00 UTC 3h 24min left n/a                         n/a          snap.certbot.renew.timer     snap.certbot.renew.service

Certbot timer appears to point at /dev/null. This question indicates that's not how it should be.

> root@aws2:/etc/systemd/system# ls -l | grep certbot
lrwxrwxrwx 1 root root    9 Jan  9 06:38 certbot.timer -> /dev/null

I can see the following in syslog but I'm not sure what it means

Mar 17 16:51:02 aws2 systemd[1]: Started Timer renew for snap application certbot.renew.

Cron

In syslog I can see this the cron job is running but there's no output

Mar 16 00:00:01 aws2 CRON[2072]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)

The following is in /etc/cron.d/certbot (this was presumably put there by the certbot installation, I get the general idea of what it does but I don't know what the test / perl stuff does)

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

When I run the whole command (root test /x etc) I get the message below.

Command 'root' not found, but can be installed with: snap install root-framework

When I run this part I get no output (note I have removed the "-q" from certbot for testing). I'm not sure what the test part is doing, but certbot doesn't seem to do anything when I run this command.

> test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot renew

Key questions

1. Any ideas what's up with the systemd timer, why it's pointing at /dev/null? Or should I just ignore this known issue?
2. Should I "snap install root-framework" to install "root" like Ubuntu is suggesting?
3. The cron job "root test" doesn't appear to be doing anything... any anyone explain what is being tested there and whether "cerbot renew" it's actually running?

Update - Proposed Solution

In the /etc/cron.daily folder I've created the following folder. I think it will do what I want, I'll check logs at some point to see. I'm still interested in the questions I asked above.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
/usr/bin/certbot -q renew

Answer 1

Since you're using snap, it's worth double checking to see if there's a second timer in place (other than certbot-renew) which is implemented by snap, and does do the renewal properly.

Here you can see I have both a mask to make sure that the normal certbot.timer is disabled, but I also have a timer installed by snap:

root@pi:/etc/systemd/system # ls -l *certbot*
lrwxrwxrwx 1 root root   9 Oct  6 10:42  certbot.timer -> /dev/null
-rw-r--r-- 1 root root 293 Oct  6 05:26  snap.certbot.renew.timer

and I can see that this runs twice daily and is currently waiting:

root@pi:/etc/systemd/system # systemctl list-units | grep certbot.renew
snap.certbot.renew.timer    loaded active waiting   Timer renew for snap application certbot.renew

root@pi:/etc/systemd/system # cat snap.certbot.renew.timer [Unit]
# Auto-generated, DO NOT EDIT 
Description=Timer renew for snap application certbot.renew 
Requires=snap-certbot-1515.mount 
After=snap-certbot-1515.mount 
X-Snappy=yes

[Timer] 
Unit=snap.certbot.renew.service 
OnCalendar=*-*-* 07:09 
OnCalendar=*-*-* 19:46

[Install] 
WantedBy=timers.target

Don't be fooled by the cronjob. It's double checking to make sure that it's not doing something that's already managed by systemd:

# grep test /etc/cron.d/certbot
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

the test used in there returns a '1' code, and hence anything after the && isn't executed.

Your workaround won't cause issues - but the stock installation should do the right thing.

So to answer your three questions::

Any ideas what's up with the systemd timer, why it's pointing at /dev/null? Or should I just ignore this known issue?

This is double checking that the snap timer is used and any system timer (e.g. when installed via apt) for certbot isn't

Should I "snap install root-framework" to install "root" like Ubuntu is suggesting?

Cron files stored in in /etc/cron.d have a syntax that include the username to be used when executing. root here is indicating that the cronjob is run as the root user. You don't need to install root-framework - the word 'root' isn't part of the command itself.

The cron job "root test" doesn't appear to be doing anything... any anyone explain what is being tested there and whether "cerbot renew" it's actually running?

As above - this cronjob is validly saying 'Hey, systemd is running! I won't do anything!'

Finally, you can verify the status of your renewals in your logfile:

# grep -c skipped /var/log/letsencrypt/letsencrypt.log
14

In my case, no renewals were happening yet (validity is long enough) but it was running - just very quietly.

Answer 2

Regarding:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

You are getting command root not found because you are including the user root with the command. The format in that cron job file goes min, hr, day, month, day of week, user, command. If you want to run it for testing just remove the username from the front of the command.

Cron jobs in general run at the set time, if the server is off during that time then the job wont run later to make up for missing it. It will only run when the "alarm" hits again if the server is on.

There is a solution to this, cron supports nicknames for run times and one of them is @reboot. Crontab(5) Manual

What i would do is setup two cron jobs, one to run on a schedule like once a week or once a month, dealers choice. Then a second job to run at @reboot to make up for anything missed while the server is off. Granted that could mean it runs daily if the server is always turned off and on. However certbot wont care, the renew program checks how close your certs are to expiring and if they are not close to expiring it wont contact the certbot server.

Unless you need it to be timers then disregard this solution...

Keep /etc/cron.d/certbot, but maybe change it to once a week instead of ever 12hrs.

# This would run every Sunday at 01:01
1 1 * * 0 root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

and add to /etc/crontab

#This would run every time the server is booted.
@reboot root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

I do not know about "snap packages" or why all that extra command line stuff is needed. Certbot in general only needs this to work.

@reboot root /usr/bin/certbot renew > /dev/null 2>&1

If you just want certbot to check for renewals once a week, assuming the server will always be online, and you don't need certbot to work in conjunction with any control panel or something for the renewals, the simplest thing to do is

Inside /etc/crontab add

1 1 * * 0 root /usr/bin/certbot renew > /dev/null 2>&1