33

When the accuracy of a DNS cache is in question, dig +trace tends to be the recommended way of determining the authoritative answer for an internet facing DNS record. This seems to be particularly useful when also paired with +additional, which also shows the glue records.

Occasionally there seems to be some disagreement on this point -- some people say that it relies on the local resolver to look up the IP addresses of the intermediate nameservers, but the command output offers no indication that this is happening beyond the initial list of root nameservers. It seems logical to assume that this wouldn't be the case if the purpose of +trace is to start at the root servers and trace your way down. (at least if you have the right list of root nameservers)

Does dig +trace really use the local resolver for anything past the root nameservers?

Andrew B
  • 31,858
  • 12
  • 90
  • 128

3 Answers3

28

This is obviously a staged Q&A, but this tends to confuse people often and I can't find a canonical question covering the topic.

dig +trace is a great diagnostic tool, but one aspect of its design is widely misunderstood: the IP of every server that will be queried is obtained from your resolver library. This is very easily overlooked and often only ends up becoming a problem when your local cache has the wrong answer for a nameserver cached.

Detailed Analysis

This is easier to break down with a sample of the output; I'll omit everything past the first NS delegation.

; <<>> DiG 9.7.3 <<>> +trace +additional serverfault.com                                                                      

;; global options: +cmd
.                   121459  IN      NS      d.root-servers.net.
.                   121459  IN      NS      e.root-servers.net.
.                   121459  IN      NS      f.root-servers.net.
.                   121459  IN      NS      g.root-servers.net.
.                   121459  IN      NS      h.root-servers.net.
.                   121459  IN      NS      i.root-servers.net.
.                   121459  IN      NS      j.root-servers.net.
.                   121459  IN      NS      k.root-servers.net.
.                   121459  IN      NS      l.root-servers.net.
.                   121459  IN      NS      m.root-servers.net.
.                   121459  IN      NS      a.root-servers.net.
.                   121459  IN      NS      b.root-servers.net.
.                   121459  IN      NS      c.root-servers.net.
e.root-servers.net. 354907  IN      A       192.203.230.10
f.root-servers.net. 100300  IN      A       192.5.5.241
f.root-servers.net. 123073  IN      AAAA    2001:500:2f::f
g.root-servers.net. 354527  IN      A       192.112.36.4
h.root-servers.net. 354295  IN      A       128.63.2.53
h.root-servers.net. 108245  IN      AAAA    2001:500:1::803f:235
i.root-servers.net. 355208  IN      A       192.36.148.17
i.root-servers.net. 542090  IN      AAAA    2001:7fe::53
j.root-servers.net. 354526  IN      A       192.58.128.30
j.root-servers.net. 488036  IN      AAAA    2001:503:c27::2:30
k.root-servers.net. 354968  IN      A       193.0.14.129
k.root-servers.net. 431621  IN      AAAA    2001:7fd::1
l.root-servers.net. 354295  IN      A       199.7.83.42
;; Received 496 bytes from 75.75.75.75#53(75.75.75.75) in 10 ms

com.                        172800  IN      NS      m.gtld-servers.net.
com.                        172800  IN      NS      k.gtld-servers.net.
com.                        172800  IN      NS      f.gtld-servers.net.
com.                        172800  IN      NS      g.gtld-servers.net.
com.                        172800  IN      NS      b.gtld-servers.net.
com.                        172800  IN      NS      e.gtld-servers.net.
com.                        172800  IN      NS      j.gtld-servers.net.
com.                        172800  IN      NS      c.gtld-servers.net.
com.                        172800  IN      NS      l.gtld-servers.net.
com.                        172800  IN      NS      d.gtld-servers.net.
com.                        172800  IN      NS      i.gtld-servers.net.
com.                        172800  IN      NS      h.gtld-servers.net.
com.                        172800  IN      NS      a.gtld-servers.net.
a.gtld-servers.net. 172800  IN      A       192.5.6.30
a.gtld-servers.net. 172800  IN      AAAA    2001:503:a83e::2:30
b.gtld-servers.net. 172800  IN      A       192.33.14.30
b.gtld-servers.net. 172800  IN      AAAA    2001:503:231d::2:30
c.gtld-servers.net. 172800  IN      A       192.26.92.30
d.gtld-servers.net. 172800  IN      A       192.31.80.30
e.gtld-servers.net. 172800  IN      A       192.12.94.30
f.gtld-servers.net. 172800  IN      A       192.35.51.30
g.gtld-servers.net. 172800  IN      A       192.42.93.30
h.gtld-servers.net. 172800  IN      A       192.54.112.30
i.gtld-servers.net. 172800  IN      A       192.43.172.30
j.gtld-servers.net. 172800  IN      A       192.48.79.30
k.gtld-servers.net. 172800  IN      A       192.52.178.30
l.gtld-servers.net. 172800  IN      A       192.41.162.30
;; Received 505 bytes from 192.203.230.10#53(e.root-servers.net) in 13 ms
  • The initial query for . IN NS (root nameservers) hits the local resolver, which in this case is Comcast. (75.75.75.75) This is easy to spot.
  • The next query is for serverfault.com. IN A and runs against e.root-servers.net., randomly selected from the list of root nameservers we just got. It has an IP address of 192.203.230.10, and since we have +additional enabled it appears to be coming from the glue.
  • Since it is not authoritative for serverfault.com, this gets delegated to the com. TLD nameservers.
  • What isn't obvious from the output here is that dig did not derive the IP address of e.root-servers.net. from the glue.

In the background, this is what really happened:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
02:03:43.301022 IP 192.0.2.1.59900 > 75.75.75.75.53: 63418 NS? . (17)
02:03:43.327327 IP 75.75.75.75.53 > 192.0.2.1.59900: 63418 13/0/14 NS k.root-servers.net., NS l.root-servers.net., NS m.root-servers.net., NS a.root-servers.net., NS b.root-servers.net., NS c.root-servers.net., NS d.root-servers.net., NS e.root-servers.net., NS f.root-servers.net., NS g.root-servers.net., NS h.root-servers.net., NS i.root-servers.net., NS j.root-servers.net. (512)
02:03:43.333047 IP 192.0.2.1.33120 > 75.75.75.75.53: 41110+ A? e.root-servers.net. (36)
02:03:43.333096 IP 192.0.2.1.33120 > 75.75.75.75.53: 5696+ AAAA? e.root-servers.net. (36)
02:03:43.344301 IP 75.75.75.75.53 > 192.0.2.1.33120: 41110 1/0/0 A 192.203.230.10 (52)
02:03:43.344348 IP 75.75.75.75.53 > 192.0.2.1.33120: 5696 0/1/0 (96)
02:03:43.344723 IP 192.0.2.1.37085 > 192.203.230.10.53: 28583 A? serverfault.com. (33)
02:03:43.423299 IP 192.203.230.10.53 > 192.0.2.1.37085: 28583- 0/13/14 (493)

+trace cheated and consulted the local resolver to obtain the IP address of the next hop nameserver instead of consulting the glue. Sneaky!

This is usually "good enough" and won't cause a problem for most people. Unfortunately, there are edge cases. If for whatever reason your upstream DNS cache is providing the wrong answer for the nameserver, this model breaks down entirely.

Real world example:

  • domain expires
  • glue is repointed at registrar redirection nameservers
  • bogus IPs are cached for ns1 and ns2.yourdomain.com
  • domain is renewed with restored glue
  • any caches with the bogus nameserver IPs continue to send people to a website that says the domain is for sale

In the above case, +trace will suggest that the domain owner's own nameservers are the source of the problem, and you're one call away from incorrectly telling a customer that their servers are misconfigured. Whether it's something you can (or are willing to) do something about is another story, but it's important to have the right information.

dig +trace is a great tool, but like any tool, you need to know what it does and doesn't do, and how to troubleshoot the issue manually when it proves insufficient.

Edit:

It should also be noted that dig +trace will not warn you about NS records that point at CNAMEaliases. This is a RFC violation that ISC BIND (and possibly others) will not attempt to correct. +trace will be completely happy to accept the A record it gets from your locally configured nameserver, whereas if BIND were to be performing full recursion it would be rejecting the entire zone with a SERVFAIL.

This can be tricky to troubleshoot if glue is present; this will work just fine until the NS records are refreshed, then suddenly break. Glueless delegations will always break BIND's recursion when a NS record points at an alias.

Community
  • 1
Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • What about `+nssearch`? – vonbrand Feb 27 '13 at 13:14
  • @vonbrand `+nssearch` performs a `NS` record lookup against your local resolver for the requested record, followed by a series of `A`/`AAAA` lookups against the local resolver for each of the returned nameservers. It's likewise susceptible to bogus nameserver records in cache. – Andrew B Feb 27 '13 at 16:50
  • 1
    So what's the solution? Use "dig ... @server" and follow the delegation manually? – Raman Sep 19 '15 at 00:12
  • @Raman Yes, it's either that or you have to empty the cache of a recursive server that you have handy, make the query, and dump the cache. (which defeats the idea of a lightweight client) dig is doing this to exponentially reduce the number of queries required. – Andrew B Sep 19 '15 at 00:19
  • _"+trace cheated and consulted the local resolver to obtain the IP address of the next hop nameserver instead of consulting the glue. Sneaky!"_ I'm wondering why this would be the default behavior and why there's no option to override it (i.e. have dig use previous answers for subsequent queries)? I'm also confused about why dig disables recursion when _+trace_ is used without providing a way of reenabling it (_+recurse_ and _-t A_ don't work). I'm assuming these are design decisions since the behavior hasn't changed in several years. – catanman Feb 14 '22 at 15:10
11

Another way of tracing DNS resolution without using the local resolver for anything except finding the root nameservers, is using dnsgraph (Full disclosure: I wrote this). It has a command line tool and a web version, of which you can find an instance at http://ip.seveas.net/dnsgraph/

Example for serverfault.com, which actually has a DNS problem right now:

enter image description here

Dennis Kaarsemaker
  • 18,793
  • 2
  • 43
  • 69
  • 5
    The stuffy pedant in me wants to say that this technically isn't an answer. The DNS admin in me thinks it's awesome and *totally doesn't care*. – Andrew B Apr 27 '14 at 10:37
  • I was going to post it as a comment, but wanted to include the image. Feel free to merge it into your answer if you think that's better. – Dennis Kaarsemaker Apr 27 '14 at 10:41
  • 1
    I'm fine with things as they are. If a mod feels otherwise I'll consolidate though, sure. – Andrew B Apr 27 '14 at 17:48
0

Very late to this thread, but I think the part of the question as to why a dig +trace uses recursive queries to local resolvers hasn't been directly explained, and this explanation is relevant to the accuracy of dig +trace's results.

After the initial recursive query for the NS records of the root zone, then dig may issue subsequent queries to local resolvers under the following conditions:

  1. a referral response is truncated due to the size of the response exceeding 512 bytes for the next iterative query

  2. dig selects an NS record from the AUTHORITY section of the referral response for which the corresponding A record (glue) is missing in the ADDITIONAL section

Because dig has only a domain name from the NS record, dig must resolve the name to an IP address by querying the local DNS server. This is the root cause (pun intended, sorry).

AndrewB has an example which is not fully consonant with what I just described, in that the root zone NS record chosen:

. 121459 IN NS e.root-servers.net.

has a corresponding A record:

e.root-servers.net. 354907 IN A 192.203.230.10

Note however that there is not a corresponding AAAA record for e-root, as well as no AAAA record for some other root servers.

Also, note the size of the response:

;; Received 496 bytes from 75.75.75.75#53(75.75.75.75) in 10 ms

496 bytes is a common size for responses that have been truncated (i.e. the next glue record would have been > 16 bytes, putting the response over 512 bytes). In other words, in a query for the NS records of root, a complete AUTHORITY and complete ADDITIONAL (both A and AAAA records) will exceed 512 bytes, so any UDP-based query which does not specify a larger query size via EDNS0 options will get a response that is cut off somewhere in the ADDITIONAL section, as the above trace shows (only f, h, i, j, and k have A and AAAA glue records).

The lack of a AAAA record for e.root-servers.net and the size of the response to the "NS ." query strongly suggest that the next recursive query was done for the reason I'm claiming. Perhaps the client O/S is IPv6-capable, and prefers AAAA records--or some other reason.

But in any case, after reading this thread, I looked into the phenomenon of dig +trace performing recursive queries subsequent to the initial one for root. The correspondence between selecting an NS record without a corresponding glue A/AAAA record and dig then sending a recursive query for that record to the local DNS is 100%, in my experience. And the reverse is true--I haven't seen recursive queries when the NS record selected from the referral has a corresponding glue record.

Binky
  • 330
  • 2
  • 11