I have twp name servers running PowerDNS as ns1 and ns2 using a master/slave-type setup, and this works well. I want to create multiple servers to provide services (i.e. web, email, etc), then have PowerDNS on either ns1 or ns2 (whichever is hit) return one of several IPs associated with a host name depending on whether that host is actually up and running.
For instance, let's say that I have two web servers, X and Y. I then have two A records in DNS so that webserver.local sometimes resolves to X's IP and sometimes to Y's IP, resulting in a round-robin effect. Perfect.
Now if X goes down, this will still work, but the client will have to time out before they get a chance to hit Y, which means that ultimately they'll get to where they want to go, but just much more slowly. If X goes down, I'd like it's IP address to be removed from the round-robin list of IPs to resolve to so that DNS would only be returning Y's IP when resolving webserver.local, until such time that X is repaired and put back into service.
Asking PowerDNS to verify that the host is up and running on each request could take a second or two (if it's even possible), which would really slow down DNS queries. One thought I've had would be to have a completely separate task periodically through cron check the servers, and modify the PowerDNS configs accordingly. I can certainly attack it from this perspective, but was hoping there might be a more standard way of approaching this that wouldn't require as much custom development.