14

So our DNS provider, every so often, experiences DDOS attacks on their systems that causes our front-facing web sites to go down.

What are some options in terms of reducing dependency on a SINGLE external managed DNS provider? My first thought was using lower expire TTL and other SOA TTLs, but it feels like these affect secondary DNS server behavior more than anything else.

i.e. If you experience a DNS outage (due to DDOS, in this example) that lasts more than, say, 1 hour, delegate everything to a secondary provider.

What do people do out there when it comes to their external DNS and using another managed DNS provider as backup?

Note to our friendly moderators: this question is much more specific then the " "generic mitigate DDOS attack" questions out there.

EDIT: 2016-05-18 (A few days later): So, first off thank you AndrewB for your excellent answer. I have some more information to add here:

So we reached out to another DNS service provider and had a chat with them. After thinking and doing a bit more research, it's actually a LOT more complicated than I thought to go with two DNS providers. This is not a new answer, it's actually more meat/info to the question! Here is my understanding:

-- A lot of these DNS providers offer proprietary features like 'intelligent DNS', for example DNS load balancing with keepalives, logic chains to configure how responses are handed back (based on geo location, various weights to records, etc. etc.). So the first challenge is to keep the two managed providers in sync. And the two managed providers are going to have to be kept in sync by the customer who has to automate interacting with their APIs. Not rocket science, but an ongoing operational cost that can be painful (given changes on both sides in terms of features and APIs).

-- But here is an addition to my question. Let's say someone did do use two managed providers as per AndrewB's response. Am I correct in that there is no 'primary' and 'secondary' DNS here as per spec? I.e., you register your four DNS server IPs with your domain registrar, two of them are one of your DNS providers, two of them are DNS servers of the other. So you would essentially just be showing the world your four NS records, all of which are 'primary'. So, is the answer to my question, "No"?

Emmel
  • 211
  • 2
  • 8
  • 2
    Who are you using as your DNS provider? Honestly, I'd switch to a different provider if this is a frequent problem, and if the provider doesn't show any signs of being able to avoid these issues. – EEAA May 16 '16 at 16:22
  • I don't want to call them out here. :-/ They're fantastic aside from this issue! – Emmel May 16 '16 at 16:26
  • 10
    Well, providing a highly-available solution is a core competency for a DNS vendor. – EEAA May 16 '16 at 16:28
  • Some hardware product exist that can help if you host yourselft, but we fall into opinion based, but you should tell your vendor honestly if you like them, maybe that will push them to invest into their structure if they care about their customers, I always told it, it's always how you bring your point that is important. – yagmoth555 May 16 '16 at 16:34
  • 2
    note that all of the big cloud providers (amazon,google,microsoft) deal with this all the time. Migrating to one of those should be option 1 – Jim B May 16 '16 at 17:50
  • CloudFlare is another option, they do DNS, WAF, and DDOS mitigation. They even have a free tier, but it's slightly limited - for example you can't enable older protocols like SSL unless you pay for a business tier. – Tim May 16 '16 at 19:47
  • Hi Emmel. In the future, can you please leave a comment if you're requesting follow up information? We don't get notifications when you update the question. I had no idea that you had posed a follow-up question in an edit, and this question is still not marked as answered. – Andrew B Aug 10 '16 at 13:29

2 Answers2

25

First, let's address the question in the title.

Is it possible to have a secondary managed DNS provider to quickly delegate to

"Quick" and "delegation" do not belong in the same sentence together when we're talking about the delegation for the top of the domain. The nameservers operated by the top level domain (TLD) registries typically serve up referrals that have TTLs measured in days. The authoritative NS records that live on your servers may have lower TTLs that end up replacing the TLD referrals, but you have no control over how often companies on the internet choose to drop their entire cache or restart their servers.

Simplifying this, it's best to assume that it's going to take at least 24 hours for the internet to pick up a nameserver change for the top of your domain. With the top of your domain being the weakest link, that's what you have to plan around the most.

What are some options in terms of reducing dependency on a SINGLE external managed DNS provider?

This question is much more solvable, and contrary to popular opinion the answer isn't always "find a better provider". Even if you use a company with a very good track record, recent years have demonstrated that no one is infallible, not even Neustar.

  • Large, well-established DNS hosting companies with good reputations are harder to crush, but bigger targets. They are less likely to go dark because someone is trying to take your domain offline, but more likely to be taken offline because they host domains that are more appealing targets. It may not happen frequently, but it still happens.
  • On the opposite extreme, running your own nameservers means that you're less likely to be sharing nameservers with a target that is more appealing than you, but it also means that you're much easier to take down if someone decides to target you specifically.

For most people, option #1 is the safest option. An outage may only happen once every few years, and if an attack does happen, it will be dealt with by people who have more experience and resources to deal with the problem.

That brings us to the final, most reliable option: a mixed approach using two companies. This provides resiliency against the problems that come with having all of your eggs in one basket.

For the sake of the argument, let's assume that your current DNS hosting company has two nameservers. If you add two nameservers managed by another company into the mix, then it takes a DDoS against two different companies to bring you offline. This will protect you against even the rare event of a giant like Neustar taking a dirt nap. The challenge instead becomes finding a way to reliably and consistently deliver updates for your DNS zones to more than one company. Typically this means having an internet facing hidden master that allows a remote partner to perform key based zone transfers. Other solutions are certainly possible, but I'm personally not a fan of using DDNS to fulfill this requirement.

The cost of the most reliable form of DNS server availability is, unfortunately, more complexity. Your problems are now much more likely to be the result of problems which cause these servers to become out of sync. Firewall and routing changes that break the zone transfers are the most common problems. Worse, if a zone transfer problem goes unnoticed for a long period of time, the expiry timer defined by your SOA record may be reached and the remote servers will drop the zone entirely. Extensive monitoring is your friend here.


To wrap all of this up, there are a number of options, and each have their drawbacks. It's up to you to balance reliability against the respective tradeoffs.

  • For most, it's enough to have your DNS hosted with a company who has a great reputation for dealing with DDoS attacks...the risk of going down once every few years is good enough for the simplicity.
  • A company with a less iron-clad reputation for dealing with DDoS attacks is the second most common option, especially when one is looking for free solutions. Just remember that free typically means no SLA guaruntee, and if a problem does happen you'll have no way to drive urgency with that company. (or a person to sue, if your legal department requires that sort of thing)
  • The least common option is, ironically enough, the most robust option of using multiple DNS hosting companies. This is due to cost, operational complexity, and perceived long term benefits.
  • The worst, at least in my opinion, is deciding to host your own. Few companies have experienced DNS admins (who are less likely to create accidental outages), experience and resources for dealing with DDoS attacks, willingness to invest in a design meeting the criteria outlined by BCP 16, and in most scenarios a combination of all three. If you want to play around with authoritative servers that only face the inside of your company, that's one thing, but internet facing DNS is an entirely different ballgame.
Andrew B
  • 31,858
  • 12
  • 90
  • 128
  • Downvote reasoning, please? – Andrew B May 16 '16 at 17:50
  • The cost of the most reliable DNS Provicer... is 0 ;) At least I never experienced any problems with CloudFlare DNS. – TomTom May 16 '16 at 17:51
  • 4
    @TomTom This isn't a few years ago. Most of the big names own at least one outage at this point. [(Cloudflare)](https://blog.cloudflare.com/todays-outage-post-mortem-82515/) ([Neustar](http://blog.catchpoint.com/2015/10/16/netflix-dns-outage/)) – Andrew B May 16 '16 at 17:52
  • *Let's say someone did do use two managed providers as per AndrewB's response. Am I correct in that there is no 'primary' and 'secondary' DNS here as per spec? I.e., you register your four DNS server IPs with your domain registrar, two of them are one of your DNS providers, two of them are DNS servers of the other. So you would essentially just be showing the world your four NS records, all of which are 'primary'.* - The concept of primary and secondary only exists between the auth servers themselves. To the outside world there is no distinction. It is common for the master to not have a `NS`. – Andrew B Aug 10 '16 at 14:20
3

There are clearly things that a DNS service provider should do, and many more that they could do, to ensure that the service is as reliable as possible.

If it appears that the service provider has unreasonable problems it would probably make sense to consider replacing them altogether but then there are also classes or problems where having separately operated services is helpful in and of itself.

As a customer, I think the most obvious option for going beyond relying on one provider would probably be to hedge your bets by having your domain(s) delegated to nameservers from multiple DNS service providers at all times (rather than changing the delegation in case of trouble).

What needs to be dealt with for that to work is essentially just keeping the zone data in sync across the nameservers of these differents providers.

The classic solution to that would be to simply use the master/slave zone transfer functionality that is part of the DNS protocol itself (this obviously requires services that allow you to make use of these facilities), either having one of the service providers be the master or possibly running your own master server.

Håkan Lindqvist
  • 33,741
  • 5
  • 65
  • 90