45

... to compensate for broken DNS servers that are outside our control.

Our problem: We deploy embedded devices that collect sensor data at various, mostly IPv4-only sites. Some sites have poorly maintained networks, e.g. misconfigured or otherwise broken DNS caches and/or firewalls that either ignore AAAA queries altogether, or respond to them with broken replies (e.g. wrong source IP!). As an external supplier to the facilities department, we have next to no influence on the (sometimes reluctant) IT departments. The chances of them fixing their DNS servers/firewalls any time soon are minuscule.

The effect on our device is that with each gethostbyname(), the processes have to wait until the AAAA queries time out, at which point some processes have already timed out their connection attempts altogether.

I am looking for solutions that are ...

  • system-wide. I cannot reconfigure dozens of application individually
  • non-permanent and configurable. We need to (re-)enable IPv6 where/when it gets fixed/rolled out. Reboot is OK.
  • If a solution requires a core library like glibc to be replaced, the replacement library package should be available from a known-to-be-well-maintained repository (e.g. Debian Testing, Ubuntu universe, EPEL). Self-building is not an option for so many reasons that I don't even know where to begin with, so I just don't list them at all...

The most obvious solution would be to configure the resolver library e.g. via /etc/{resolv,nsswitch,gai}.conf to not query AAAA records. A resolv.conf option no-inet6 as suggested here would be exactly what i am looking for. Unfortunately it is not implemented, at least not on our systems (libc6-2.13-38+deb7u4 on Debian 7; libc6-2.19-0ubuntu6.3 on Ubuntu 14.04)

So how then? One finds the following methods suggested on SF and elsewhere, but non of them work:

  • Disabling IPv6 altogether, e.g. by blacklisting the ipv6 LKM in /etc/modprobe.d/, or sysctl -w net.ipv6.conf.all.disable_ipv6=1. (Out of curiosity: Why is the resolver asking for AAAA where IPv6 is disabled?)
  • Removing options inet6 from /etc/resolv.conf. It wasn't there in the first place, inet6 is simply enabled by default these days.
  • Setting options single-request in /etc/resolv.conf. This only ensures that the A and the AAAA queries are done sequentially rather than in parallel
  • Changing precedence in /etc/gai.conf. That does not affect the DNS queries, only how multiple replies are processed.
  • Using external resolvers (or running a local resolver daemon that circumvents the broken DNS servers) would help, but is usually disallowed by the company's firewall policies. And it can make internal resources inaccessible.

Alternative ugly ideas:

  • Run a DNS cache on localhost. Configure it to forward all non-AAAA queries, but to respond to AAAA queries with either NOERROR or NXDOMAIN (depending on the result of the corresponding A-query). I am not aware of a DNS cache able to do this though.
  • Use some clever iptables u32 match, or Ondrej Caletka's iptables DNS module to match AAAA queries, in order to either icmp-reject them (how would the resolver lib react to that?), or to redirect them to a local DNS server that responds to everything with an empty NOERROR.

Note that there are similar, related questions on SE. My question differs insofar as it elaborates the actual problem i am trying to solve, as it lists explicit requirements, as it blacklists some often-suggested non-working solutions, and as it is not specific to a single application. Following this discussion, I posted my question.

Nils Toedtmann
  • 3,202
  • 5
  • 25
  • 36
  • 16
    PS: Contrary to popular belief here on SF, there are some good reasons to disable IPv6/AAAA on a machine in a IPv4-only network, even where DNS works: Reduce broadcast load; Reduce load on DNS resolvers by almost 50%; Reduce connection start-up times (significantly where DNS caches are laggy); Follow best practices to disable non-functional features to enhance security and stability. Admittedly, if i forget to re-enable IPv6 once it becomes available, then my system becomes IPv4 legacy ballast that impedes IPv6 rollout. One should be allowed to weigh the listed pros against this con. – Nils Toedtmann Oct 01 '14 at 09:33
  • Any reason why you don't run a full resolver on localhost? That way you eliminate the dependency on other people's (seemingly) unreliable DNS resolvers altogether. – Sander Steffann Oct 01 '14 at 09:47
  • @SanderSteffann Company firewall policies usually disallow that. But elsewhere that is an option. I'll add it to my question later. – Nils Toedtmann Oct 01 '14 at 09:56
  • So you build a device assuming full IPv6 capability at your customer networks and then blame your customer when they don't have full IPv6 capability? It seems like the fault is yours not theirs. On top of that you're going to bad mouth your customers by calling them clueless. If you were one of my vendors you'd be on the short list for being replaced. – joeqwerty Oct 01 '14 at 12:35
  • 3
    @joeqwerty We are not making any assumptions about whether or not IPv6 is supported on site. We do make the assumption though that DNS servers are standard-compliant. Also, some IT departments unfortunately *do* lack the skills to configure their infrastructure properly. Sorry for being blunt about that. – Nils Toedtmann Oct 01 '14 at 12:50
  • 4
    I understand what you're saying. You need to make your box work on their network not the other way around, and that is what you're here asking about. I just get a little miffed when we in the IT field blame our customers and don't respect them. They are our bread and butter. For good or bad we need to respect that and respect them. Our customers are not a hindrance to our business, they are the reason for our business. – joeqwerty Oct 01 '14 at 13:01
  • Any chance of adjusting the timings of either the lookups or the processes that depend on the lookups? – joeqwerty Oct 01 '14 at 13:04
  • There is `options timeout:X` in /etc/resolv.conf which i could reduce from 5 to say 2/3 seconds. That might ease my problem a bit. I can't set the global timeouts for A and AAAA separately. BTW i eased my wording. – Nils Toedtmann Oct 01 '14 at 13:08

5 Answers5

12

Stop using gethostbyname(). You should be using getaddrinfo() instead, and should have been for years now. The man page even warns you of this.

The gethostbyname*(), gethostbyaddr*(), herror(), and hstrerror() functions are obsolete. Applications should use getaddrinfo(3), getnameinfo(3), and gai_strerror(3) instead.

Here is a quick sample program in C which demonstrates looking up only A records for a name, and a Wireshark capture showing that only A record lookups went over the network.

In particular, you need to set ai_family to AF_INET if you only want A record lookups done. This sample program only prints the returned IP addresses. See the getaddrinfo() man page for a more complete example of how to make outgoing connections.

In the Wireshark capture, 172.25.50.3 is the local DNS resolver; the capture was taken there, so you also see its outgoing queries and responses. Note that only an A record was requested. No AAAA lookup was ever done.

#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <stdlib.h>
#include <netdb.h>
#include <stdio.h>

int main(void) {
    struct addrinfo hints;
    struct addrinfo *result, *rp;
    int s;
    char host[256];

    memset(&hints, 0, sizeof(struct addrinfo));
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_protocol = 0;

    s = getaddrinfo("www.facebook.com", NULL, &hints, &result);
    if (s != 0) {
        fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s));
        exit(EXIT_FAILURE);
    }

    for (rp = result; rp != NULL; rp = rp->ai_next) {
        getnameinfo(rp->ai_addr, rp->ai_addrlen, host, sizeof(host), NULL, 0, NI_NUMERICHOST);
        printf("%s\n", host);
    }
    freeaddrinfo(result);
}
Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • 2
    Interesting! I will investigate which applications trigger AAAA requests. If it's only ours, then i'll relay your suggestion to our devs. But i have a strong hunch that a lot of Debian/Ubuntu-packaged software is suffering from this, and we ain't gonna patch those. – Nils Toedtmann Jul 27 '15 at 09:13
  • Your application is probably the most important one. Even if you can't fix everything else, it may improve the situation. – Michael Hampton Jul 27 '15 at 13:24
  • 2
    That doesn't help when the call gethostbyname is in code you don't control. – Thayne Apr 28 '21 at 00:52
  • @Thayne That's true, but at this point getaddrinfo() is 20 years old, so if your business still depends on a binary of that vintage you've almost certainly got more serious problems. – Michael Hampton Apr 30 '21 at 17:15
  • 2
    @MichaelHampton : Both `curl` and `wget` apparently suffer from this. Good luck getting an even moderately complex linux system to work, if neither of these work reliably. – Peter V. Mørch Jun 18 '21 at 09:38
  • While this might address the OP's base need, it completely bypasses the actual text of the question, which is how to disable aaaa lookups. And it's no longer true. – Kurt Fitzner Apr 21 '22 at 19:41
5

When in doubt, head over to the source code! So, let's see... gethostbyname() looks interesting; that describes exactly what we're seeing: try IPv6 first, then fall back to IPv4 if you don't get an answer you like. What's this RES_USE_INET6 flag? Tracing it back, it's coming from res_setoptions(). This is where resolv.conf is read in.

And.... that's me out of ideas. I'm completely unclear how it is that RES_USE_INET6 is being set if not in resolv.conf.

BMDan
  • 7,129
  • 2
  • 22
  • 34
  • RES_USE_INET6 *can* be set via `options inet6` in `resolv.conf`. I guess my problem is that it cannot be *unset* once it was set at compile time, which all major distros seem to do these days (correct?). Therefore the feature request for `options no_inet6` that i mentioned above. – Nils Toedtmann Oct 03 '14 at 07:03
  • 1
    Unfortunately, as you can see from the code, there doesn't appear to be a `no_inet6` option in `res_setoptions()`. However, as you can see from (no-)`ip6-dotint`, it's an easy change to add. To test the theory that it's being set by default by your distro, I'd grab the package source files and compile it once "virgin" (to confirm that the package replicates the behavior) and then add: `{ STRnLEN ("no-inet6"), 1, ~RES_USE_INET6 },` to the `options[]` array and see if the problem goes away when you set that option in `resolv.conf`. – BMDan Oct 06 '14 at 21:24
  • 2
    Lastly: for what it's worth, I'd solve this by running a DNS cache on localhost (as you reference, above). It'd be a heckuva lot easier to maintain your own, hacked DNS proxy/cache than it would be to maintain a hacked version of a core system library. – BMDan Oct 06 '14 at 21:26
3

You could use BIND as a local resolver, it has an option to filter AAAA:

https://kb.isc.org/article/AA-00576/0/Filter-AAAA-option-in-BIND-9-.html

  • 3
    That's pretty heavyweight for an embedded device. – Michael Hampton Jul 25 '15 at 21:25
  • Thank you, i wasn't aware of Bind's AAAA filter. As Michael mentions, it's probably not a solution for us due to Bind's large footprint. But for those who want to filter out AAAA responses in other scenarios it might be a viable way. Ubuntu actually builds bind with "--enable-filter-aaaa", at least on 14.04. Not sure about Debian. - See also http://ipamworldwide.blogspot.co.uk/2011/09/handy-aaaa-filter-in-bind-98.html – Nils Toedtmann Jul 26 '15 at 20:20
  • 1
    I am on 14.04 and it does not seem that this filtering option is available. – Zitrax Apr 06 '17 at 15:04
0

You can use CoreDNS as local resolver with aaaa plugin. This is a custom plugin which needs coredns plugin.cfg to be modified and then you need to rebuild it (see here for more info), or you can use this fork which is compiled with aaaa plugin. (There is a docker image in ghcr). Alternatively you can use official CoreDNS with template plugin with this config:

template AAAA {
   rcode NXDOMAIN
}

However aaaa plugin has less overhead if you have a huge DNS load.

The advantage compared to BIND is that coredns have a smaller footprint, and also you can disable many of its default plugins and rebuilt it to have even a smaller CoreDNS.

0

Did you try to setup PDNS-recursor, set it in your /etc/resolv.conf and deny "AAAA" lookups in it? Using something like query-local-address6=

Glueon
  • 3,514
  • 2
  • 22
  • 31
  • 2
    `query-local-address6=` does something different (which IPv6 address to send queries from - note that even with IPv6 disabled, AAAA requests will still be resolved via IPv4). Also I cannot identify any other setting that would filter AAAA queries (http://doc.powerdns.com/html/built-in-recursor.html). Without that information, your answer is not very helpful :( – Nils Toedtmann Oct 02 '14 at 10:40