On the inside of your network, for outbound traffic, you could run a FHRP* between your two routers. This will require you to reip device(s), as hosts will have a new default gateway. You would create or reuse a VIP [virtual ip] and assign that to the inside of both routers.
However, the protocols will, either by configuration or default settings, automatically forward traffic based on a configurable 'criteria'. Generally you're tracking connectivity to a layer 3 address (like your WAN default gateway), you'd want to consult the documentation and feature set of your routers.
The general idea is, when your FHRP, of choice, detects the internet down, it allows your 'standby' router (or active if this is needed) to begin responding to the virtual MAC ARP requests and it begins forwarding traffic with the same VIP shared by the primary router as its source. Again, protocol dependent, but some can use 'preempt' which will overthrow and revert back to the primary network once the fail condition is corrected. Some give more granular settings that can really fine tune to most specs.
Additionally, based on your traffic patterns and flow, you could load share, or session balance, between both circuits for outbound traffic. This has the benefit of ~100% outbound across two circuits (~50/~50 share) - instead of 100% across a single circuit. This is asymmetric routing, and sometimes isn't desired. It all really depends on your objectives and layout. An example of this is packets going outbound from your backup circuit, received by the client, but the replies go to your primary DNS. Out one router, in another. This could be a problem, or it could be the solution to the problem.
This doesn't take care of your secondary requirement for DNS, but it does provide the internal redundancy.
- First Hop Redundancy Protocol (VRRP/GLBP/HSRP/CARP/NSRP etc.)
Updating the DNS is the only solution I could think of and it's not a great one. Do you think ISPs have something setup to handle this? I haven't asked them. Perhaps it's possible you can give them a backup ip to forward on their end? – user190084 – 2014-12-03T23:42:59.347
Amazon's Route53, DNSMadeEasy, and OpenDNS are three DNS providers that offer something like I described above, and there are probably many others. You can ask your ISP, but odds are they can't do what is needed. – Jim G. – 2014-12-04T00:08:44.393
Also, in your second scenario what happens if the destination to the load balancer is down? – user190084 – 2014-12-04T21:21:48.403
Then everything is down. That's why it's important to set the load balancer up right. Think along the lines of an AWS ELB. You can't easily build this yourself without a lot of effort. Essentially Amazon (or any other service provider) has done some of the hard work for you. – Jim G. – 2014-12-04T22:23:24.743
http://www.dnsmadeeasy.com/services/dns-failover-system-monitoring/ – user190084 – 2014-12-16T17:41:36.093