How can I use NGINX to reverse-proxy a MQTT service over SSL?

Question

I've setup several server blocks for websites with NGINX. I can reverse-proxy (as well as force all traffic to https on these sites). I'm trying to setup a MQTT broker so that it uses SSL.

server {

   server_name mqtt.example.com;
   listen 80;
   location /
   {
      proxy_pass http://172.16.68.67:1884;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
   }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mqtt.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mqtt.example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

I can use connect to my mqtt service using port 80 with http://www.hivemq.com/demos/websocket-client/ but I can't connect to port 443. That gives me a generic "socket error" message.