Am I going crazy, or is my ISP rewriting DNS A records?

Question

Here is my problem:

When connected to my FIOS ISP, with no router, I.E, directly connected to my FIOS modem, I am getting different A DNS records than I am when connected to a different ISP/Network.

1. On my FIOS ISP I run: nslookup netflix.com - this is with no dns specified, and get:

Non-authoritative answer:
Name:    netflix.com
Addresses:  2a01:578:3::22fa:2993
          2a01:578:3::3413:2893
          2a01:578:3::341f:91b7
          2a01:578:3::34d1:eb8d
          2a01:578:3::341e:2dc6
          2a01:578:3::36ab:7445
          2a01:578:3::34d2:745
          2a01:578:3::341f:b664
          54.77.143.196
          52.208.135.54
          34.252.179.162
          52.209.79.186
          52.17.227.174
          52.51.252.111
          54.171.187.60
          52.30.103.23

2. Again, on my FIOS ISP, I then run: nslookup netflix.com 85.203.37.1 with the 85.203.37.1 DNS resolver specified, and I get back exactly the same thing as (1):

Non-authoritative answer:
Name:    netflix.com
Addresses:  2a01:578:3::22fa:2993
          2a01:578:3::3413:2893
          2a01:578:3::341f:91b7
          2a01:578:3::34d1:eb8d
          2a01:578:3::341e:2dc6
          2a01:578:3::36ab:7445
          2a01:578:3::34d2:745
          2a01:578:3::341f:b664
          54.77.143.196
          52.208.135.54
          34.252.179.162
          52.209.79.186
          52.17.227.174
          52.51.252.111
          54.171.187.60
          52.30.103.23

3. However, now what is totally boggling my mind, if I run nslookup netflix.com 85.203.37.1 with the 85.203.37.1 DNS resolver specified, on any other network, that is through a VPN, or on mobile devices, etc. I get back

Non-authoritative answer:
Name:    netflix.com
Addresses:  198.255.83.3
          107.182.237.252

(3) happens to be the response I am expecting to get back, what I don't understand is how, when connected to my FIOS ISP in (2), it's almost as though I have not specified a DNS resolver, even though I have, and how the A DNS records are exactly the same as the ones which my ISP's DNS returns.

The thing that makes this really weird for me is that, this seems like some kind of DNS interception or hijacking. And after doing, a lot of Googling about DNS interception, it's not clear to me that it is actually occuring.

Annoyingly, I can't find one consistent test which will always prove that DNS request are being intercepted, I guess because there are so many different ways to intercept or redirect DNS requests. Some of the 'tests' I have done are these:

1. https://padlock.argh.in/2019/04/28/sky-dns-interception.html
2. https://superuser.com/a/1348765/450105
3. https://labs.ripe.net/Members/babak_farrokhi/is-your-isp-hijacking-your-dns-traffic

In all of these tests, I don't seem to get any of the intercepting-is-happening results.

I have tried to learn as much as I can about the specifics of DNS interception, but it is so complicated, that I can't say I understand it. But, given the information I have collected, the only explanation I can think of is that my FIOS ISP is somehow rewriting A DNS records.

Is this a reasonable conclusion, or is there something I have missed?

---

I have put the disclaimer that I might be going crazy, because this problem is slowly driving me insane .

Answer 1

You are not going crazy, and your ISP is not tampering with your DNS results. The modern Internet is just far more complex and virtualized than you might be assuming it to be.

High-traffic websites use Content Delivery Networks (CDNs). CDNs have servers distributed all around the "edges" of the Internet; that is, close to where consumers connect to the Internet. That way, when you go to stream something from Netflix, it's not coming from Netflix headquarters in Los Gatos, California, USA, it's coming from a CDN "edge node" server attached to your local ISP's network infrastructure. That way the response time, bandwidth, and reliability are all much better than if was streaming from so far away.

When you do a DNS query for a CDN-hosted domain name, the CDN's DNS servers answer you with A and AAAA records pointing to their edge nodes that are closest to you, based on the IP address that your DNS query came from.

Okay, but most resolving DNS servers don't leak your client's IP address when they recursively resolve a DNS query for you. So why are you getting different answers from the same DNS server when you're connecting from different networks?

Well, in your case, it seems the DNS server you're using (85.203.37.1) is hosted in IBM cloud hosting via a "Bring Your Own IP" (BYOIP) arrangement, so even your traffic to 85.203.37.1 is actually going to different physical servers depending on which network your traffic is coming from. So even though that IP address is from a netblock assigned to falco-networks.com, it seems that Falco has arranged for IBM to "own" the route to that netblock, so traffic to that IP address gets routed to the nearest entry point to IBM's global network, and then IBM routes it to their nearest edge node.

So when I traceroute to 85.203.37.1, it terminates in my own metropolitan area in the US, even though that netblock is part of RIPE's (the European Regional Internet Registry) IPv4 allocation, and even though Falco seems to be a company from the Netherlands. Without a bunch of network virtualization going on via CDNs and cloud services, I would never expect that traceroute to terminate anywhere but Europe. I suspect that when you traceroute to it, it'll terminate somewhere in the Netherlands for you (I'm guessing you're in the Netherlands).

If you want to double-check me on this with your own traceroutes and whois queries, note that the domains "networklayer.com", "softlayer.net", and "sl-reverse.com" all appear to be owned by IBM and used as part of their cloud infrastructure. So if you see any of those names in the last or next-to-last named hop, it's a strong indication that your traffic has entered IBM's cloud services network at that point.