How to migrate letsencrypt renewal from apache2 to nginx

Question

I inherited a very new magento configuration from a previous employee (who left for another job) where I currently work. The original Magento was set up with v2.1.8 but I have had to update/upgrade the installation a number of times as the needs/requirements of the project have changed. As a result, I do not necessarily have all the original files as they were installed (I backed up some, but after changing multiple times, I had to trim down what I was backing up) and can't necessarily ask the person who originally installed/configured this what he did.

The original set-up was just a stand-alone magento-CE on apache2 with https. Part of the problem is that I am now using the Magento advice and set up apache to just do http in the background with a varnish cache and nginx handling the actually https stuff.

The first problem occurred when the apache server was mysteriously shutting down overnight with no fan-fare. Just a received SIGTERM message in the error log which suggested something was shutting it down. I traced that down to a cron job running a renewal script for letsencrypt that was trying to verify the certificate via apache2 (which is no longer doing https or talking on port 443)

What I need now is information on how to finish migrating the set-up so it the renewal script will work with nginx instead of apache2. All of the instructions I can find on setting up letsencrypt seem to assume you haven't done anything yet and don't yet have a certificate. But I do have certificates, I just don't have whatever nginx and letsencrypt need to run the renewal properly.

I modified the /etc/letsencrypt/renewal/myhost.conf to use nginx instead of apache for the authenticator. (I also modified the 'installer' line but not sure if that is needed since it was installed under apache2)

I recreated a .well-known/acme-challenge/test file in the server root and added a Directory and Location issue and confirmed it accessible from both the apache server on http at port 8080 and through the cache on the nginx on port 443 with https. (returning 'success' as the content)

But when I run certbot to test the renewal (per suggestions I saw in the various instruction pages) it fails:

# certbot renew --pre-hook "service nginx stop" -
-post-hook "service nginx start"
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/www.mydomain.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.mydomain.com
tls-sni-01 challenge for mydomain.com
nginx: [error] open() "/run/nginx.pid" failed (2: No such file or directory)
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (www.mydomain.com) from /etc/letsencrypt/renewal/www.mydomain.com.conf produced an unexpected error: Failed authorization procedure. www.mydomain.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.mydomain.com/fullchain.pem (failure)

-------------------------------------------------------------------------------

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.mydomain.com/fullchain.pem (failure)
-------------------------------------------------------------------------------
Running post-hook command: service nginx start
Hook command "service nginx start" returned error code 1
Error output from service:
Job for nginx.service failed because the control process exited with error code$ See "systemctl status nginx.service" and "journalctl -xe" for details.

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.mydomain.com
   Type:   connection
   Detail: Timeout

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

I need to know what is still missing to get the renewal to work properly

Answer 1

I did this for 3 domains now and it worked, but try it at your own risk and make backups of original files before you do anything.

• Make sure you actually have the nginx plugin. You can install it with the command sudo apt-get install python-certbot-nginx
• Edit /etc/letsencrypt/renewal/*.conf with an editor of your choice
  • I did vim /etc/letsencrypt/renewal/DOMAIN.conf
• In each of these files, there are two lines where you need to replace apache2 with nginx
  • authenticator: nginx
  • installer: nginx

Finally, to test if it went well, try renewing your certs with

certbot --dry-run renew

If you don't get any errors, it means the renewal succeeded.

Answer 2

You can try to follow another approach using universal authenticator --standalone instead of switching between Apache or Nginx. However you will face an issue as they say: "This plugin needs to bind to port 80 in order to perform domain validation, so you may need to stop your existing webserver.". So, in case you have to renew certificate for the webserver it gets downtime for the period of domain validation.

There is a solution that can help you to avoid downtime:

1. Run certbot with parameters: --standalone --http-01-port 12346 — the plugin's webserver will be launched on port 12346 instead of 80, so you won't need to stop your webserver.

2. Run intermediate small reverse proxy for instance Tinyproxy listening to port 12345.

   This proxy is very lightweight and easy to manage, so don't be afraid. It should listen to HTTP traffic and redirect requests between two webservers. Proxy filters traffic and redirects a Let's Ecnrypt validation request to the plugin's webserver to port 12346 and regular HTTP requests to your webserver.

3. Create HTTP traffic redirection to reverse proxy from port 80 to port 12345 with iptables and accept traffic on new ports:

   iptables -I INPUT -p tcp -m tcp --dport 12346 -j ACCEPT
   iptables -I INPUT -p tcp -m tcp --dport 12345 -j ACCEPT
   iptables -t nat -I PREROUTING -p tcp -m tcp ! -s 127.0.0.1/32 --dport 80 \
       -j REDIRECT --to-ports 12345
   

4. Right after validation is complete remove firewall rules and stop the proxy server.

That's all you got automatic Let's Encrypt certificate renewal with no downtime and no changes of your webserver content.

You may get familiar with this recipe here: https://jelastic.com/blog/zero-downtime-renewal-letsencrypt-ssl-certificate/

Answer 3

I managed to get it working by:

1. setting the authenticator to nginx
2. commenting out the pre_hook script
3. editing the post_hook script by changing systemctl start nginx to systemctl restart nginx in /etc/letsecrypt/renewal/*.com.conf.

Answer 4

I have just done the move for my personal server. My server is hosted in Digital Ocean but generally you can do that with any server hosted anywhere.

You just need to configure new blocks for nginx containing your domains, after that calling certbot

sudo certbot --nginx -d example.com -d www.example.com

If you also want to migrate the certificates to a new server, I summed everything up in my blog here https://peacemoon.de/blog/2019/01/13/moving-servers-and-lets-encrypt-certificates-from-apache-to-nginx/