A domain can have plenty of name servers registered at there domain registrar. The name servers are picked randomly and not like expected primary first, secondary second and so on.
Knowing that, does this mean that when one name server is down, there is a 50% chance that the visitors that get to question the offline name server will never reach your site? While the other 50% is able to browse to your site just fine, and so affecting the availability of the server?
Lastly, why would clients not by default question the next name server in the list when one is down?
The same applies for IPv4 and IPv6. If one of the name servers only supports IPv6 and no IPv4 and a user without IPv6 connectivity gets to question that specific name server, the site will be unreachable I suppose.
Additionally, I'm talking explicitly about the way the authoritative server is picked and the handling of a failure in case the picked authoritative server is not available due to downtime or ipv4-ipv6 incompatibilities between client and server.