Bind9 nsupdate update-policy rule to grant _acme-challenge.**.domain.tld (all additional subdomains)

Question

Issue

I'm using letsencrypt certbot's DNS-01 challenge, but it won't issue certificates more than one subdomain level deep.

named.conf

# grep -A 3 ^key /etc/bind/named.conf.local
key "certbot." {
    algorithm hmac-sha512;
    secret    "[REDACTED]";
};
# grep -A 2 example.tld /etc/bind/named.conf.local
zone    "example.tld"              {
    type            master;
    file            "/var/cache/bind/fdb.example.tld.signed";
    allow-transfer  { pub-ns-acl; };
    update-policy   {
        grant   certbot. name _acme-challenge.example.tld. txt;
    };
};

Single subdomain

I know my keys are configured correctly because I can issue certs for a single subdomain, even if it's a wildcard:

# certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/rfc2136/cred.ini --preferred-challenges=dns --email=certbot@example.com --agree-tos -d *.example.tld -d example.tld --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate

IMPORTANT NOTES:
 - The dry run was successful.

Double subdomain

This is where it doesn't work.

# certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/rfc2136/cred.ini --preferred-challenges=dns --email=certbot@ki9.us --agree-tos -d example.tld -d *.example.tld -d www.www.example.tld --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None                                                                                           

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -                                                                       
You have an existing certificate that contains a portion of the domains you requested (ref: /etc/letsencrypt/renewal/example.tld.conf)                                                                                               

It contains these names: *.example.tld, example.tld

You requested these names for the new certificate: *.example.tld, example.tld, www.www.example.tld.

Do you want to expand and replace this existing certificate with the new certificate?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(E)xpand/(C)ancel: E
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for www.www.example.tld
Cleaning up challenges
Encountered exception during recovery: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 139, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 76, in _perform
    self._get_rfc2136_client().add_txt_record(validation_name, validation, self.ttl)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 135, in add_txt_record
    .format(dns.rcode.to_text(rcode)))
certbot.errors.PluginError: Received response from server: REFUSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/error_handler.py", line 108, in _call_registered
    self.funcs[-1]()
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 323, in _cleanup_challenges
    self.auth.cleanup(achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 76, in cleanup
    self._cleanup(domain, validation_domain_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 79, in _cleanup
    self._get_rfc2136_client().del_txt_record(validation_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 170, in del_txt_record
    .format(dns.rcode.to_text(rcode)))
certbot.errors.PluginError: Received response from server: REFUSED
Received response from server: REFUSED

Verbose edition:

# certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/rfc2136/cred.ini --preferred-challenges=dns --email=certbot@ki9.us --agree-tos -d example.tld -d *.example.tld -d www.www.example.tld --dry-run -vvv
Root logging level set at -10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator dns-rfc2136 and installer None
Single candidate plugin: * dns-rfc2136
Description: Obtain certificates using a DNS TXT record (if you are using BIND for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-rfc2136 = certbot_dns_rfc2136.dns_rfc2136:Authenticator
Initialized: <certbot_dns_rfc2136.dns_rfc2136.Authenticator object at 0x7fda4a6974e0>
Prep: True
Selected authenticator <certbot_dns_rfc2136.dns_rfc2136.Authenticator object at 0x7fda4a6974e0> and installer None
Plugins selected: Authenticator dns-rfc2136, Installer None
Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/12742232', new_authzr_uri=None, terms_of_service=None), [REDACTED], Meta(creation_dt=datetime.datetime(2020, 3, 11, 1, 14, 11, tzinfo=<UTC>), creation_host='localhost'))>
Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 724
Received response:
HTTP 200
Server: nginx
Date: Fri, 21 Aug 2020 15:47:02 GMT
Content-Type: application/json
Content-Length: 724
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "cwHOlqiOgc0": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
Renewal conf file /etc/letsencrypt/renewal/www.otherdomain.tld.conf is broken. Skipping.
Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/cert_manager.py", line 383, in _search_lineages
    candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 463, in __init__
    self._check_symlinks()
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 522, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/www.otherdomain.tld/cert.pem to be a symlink


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/example.tld.conf)

It contains these names: *.example.tld, example.tld

You requested these names for the new certificate: *.example.tld, example.tld,
www.www.example.tld.

Do you want to expand and replace this existing certificate with the new
certificate?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(E)xpand/(C)ancel: E
Renewing an existing certificate
Requesting fresh nonce
Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
Received response:
HTTP 200
Server: nginx
Date: Fri, 21 Aug 2020 15:47:03 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: [REDACTED]
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


Storing nonce: [REDACTED]
JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "*.example.tld"\n    },\n    {\n      "type": "dns",\n      "value": "example.tld"\n    },\n    {\n      "type": "dns",\n      "value": "www.www.example.tld"\n    }\n  ]\n}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "[REDACTED]",
  "signature": "[REDACTED]",
  "payload": "[REDACTED]"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 621
Received response:
HTTP 201
Server: nginx
Date: Fri, 21 Aug 2020 15:47:03 GMT
Content-Type: application/json
Content-Length: 621
Connection: keep-alive
Boulder-Requester: 12742232
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/[REDACTED]/[REDACTED]
Replay-Nonce: [REDACTED]
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2020-08-28T15:44:16Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "*.example.tld"
    },
    {
      "type": "dns",
      "value": "example.tld"
    },
    {
      "type": "dns",
      "value": "www.www.example.tld"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/[REDACTED]",
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/[REDACTED]",
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/[REDACTED]"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/[REDACTED]/[REDACTED]"
}
Storing nonce: [REDACTED]
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/95814309:
{
  "protected": "[REDACTED]",
  "signature": "[REDACTED]",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/[REDACTED] HTTP/1.1" 200 472
Received response:
HTTP 200
Server: nginx
Date: Fri, 21 Aug 2020 15:47:03 GMT
Content-Type: application/json
Content-Length: 472
Connection: keep-alive
Boulder-Requester: 12742232
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: [REDACTED]
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "example.tld"
  },
  "status": "valid",
  "expires": "2020-09-17T22:29:52Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "valid",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/[REDACTED]/MUKGGw",
      "token": "[REDACTED]",
      "validationRecord": [
        {
          "hostname": "example.tld"
        }
      ]
    }
  ],
  "wildcard": true
}
Storing nonce: [REDACTED]
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/[REDACTED]:
{
  "protected": "[REDACTED]",
  "signature": "[REDACTED]",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/[REDACTED] HTTP/1.1" 200 452
Received response:
HTTP 200
Server: nginx
Date: Fri, 21 Aug 2020 15:47:04 GMT
Content-Type: application/json
Content-Length: 452
Connection: keep-alive
Boulder-Requester: 12742232
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: [REDACTED]-vau0I
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "example.tld"
  },
  "status": "valid",
  "expires": "2020-09-17T22:29:52Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "valid",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/[REDACTED]/[REDACTED]",
      "token": "[REDACTED]",
      "validationRecord": [
        {
          "hostname": "example.tld"
        }
      ]
    }
  ]
}
Storing nonce: [REDACTED]
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/97803217:
{
  "protected": "[REDACTED]",
  "signature": "[REDACTED]",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/97803217 HTTP/1.1" 200 812
Received response:
HTTP 200
Server: nginx
Date: Fri, 21 Aug 2020 15:47:04 GMT
Content-Type: application/json
Content-Length: 812
Connection: keep-alive
Boulder-Requester: 12742232
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: [REDACTED]
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "www.www.example.tld"
  },
  "status": "pending",
  "expires": "2020-08-28T15:44:16Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/[REDACTED]/[REDACTED]",
      "token": "[REDACTED]"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/[REDACTED]/[REDACTED]",
      "token": "[REDACTED]"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/[REDACTED]/[REDACTED]",
      "token": "[REDACTED]"
    }
  ]
}
Storing nonce: [REDACTED]
Performing the following challenges:
dns-01 challenge for www.www.example.tld
No authoritative SOA record found for _acme-challenge.www.www.example.tld
No authoritative SOA record found for www.www.example.tld
No authoritative SOA record found for www.example.tld
Received authoritative SOA response for example.tld
Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 139, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 76, in _perform
    self._get_rfc2136_client().add_txt_record(validation_name, validation, self.ttl)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 135, in add_txt_record
    .format(dns.rcode.to_text(rcode)))
certbot.errors.PluginError: Received response from server: REFUSED

Calling registered functions
Cleaning up challenges
No authoritative SOA record found for _acme-challenge.www.www.example.tld
No authoritative SOA record found for www.www.example.tld
No authoritative SOA record found for www.example.tld
Received authoritative SOA response for example.tld
Encountered exception during recovery: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 139, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 76, in _perform
    self._get_rfc2136_client().add_txt_record(validation_name, validation, self.ttl)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 135, in add_txt_record
    .format(dns.rcode.to_text(rcode)))
certbot.errors.PluginError: Received response from server: REFUSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/error_handler.py", line 108, in _call_registered
    self.funcs[-1]()
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 323, in _cleanup_challenges
    self.auth.cleanup(achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 76, in cleanup
    self._cleanup(domain, validation_domain_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 79, in _cleanup
    self._get_rfc2136_client().del_txt_record(validation_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 170, in del_txt_record
    .format(dns.rcode.to_text(rcode)))
certbot.errors.PluginError: Received response from server: REFUSED
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1250, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 310, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 389, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 139, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 76, in _perform
    self._get_rfc2136_client().add_txt_record(validation_name, validation, self.ttl)
  File "/usr/lib/python3/dist-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 135, in add_txt_record
    .format(dns.rcode.to_text(rcode)))
certbot.errors.PluginError: Received response from server: REFUSED
Received response from server: REFUSED

Observations, potential solution

One line that caught my eye was this one:

No authoritative SOA record found for _acme-challenge.www.www.example.tld

Certbot is probably trying to nsupdate on the subdomain _acme-challenge.www.www.example.tld, but the update-policy won't allow it:

grant certbot. name _acme-challenge.example.tld. txt;

So the question is: What update-policy rule can I used to allow nsupdate on _acme-challenge.*.*.example.tld? Is there a rule that would cover further subdomaining like _acme-challenge.*.*.*.example.tld?

score 2 · Answer 1 · answered Aug 21 '20 at 17:29

After spending more than a day on it, I found it as I was writing the question.

Just as there is no wildcard expansion available past the left-most subdomain, you can't use the update-policy wildcard ruletype that way either. That is, it won't work for _acme-domain.*.example.tld, but would for *.www.example.tld.

Considering I already know that the left-most subdomain is _acme-challenge, a wildcard is unnecessary. The best I can do is to set the update-policy explicitly for all subdomains:

update-policy {
    grant certbot. name _acme-challenge.example.tld. txt;
    grant certbot. name _acme-challenge.www.example.tld. txt;
};

source: Bind9 Docs: Dynamic update policies

Yes, I think the `external` rule type is the one option that could facilitate what you had in mind in the question. However, that is not just a bit of configuration but building an external application that decides if the updates are accepted or not. — Håkan Lindqvist, Aug 21 '20 at 20:44