After my initial comment a slightly longer answer.
As a rule of thumb when the protocol itself, or the common implementations of said protocol, already natively support resilience, high availability and failover, you don't need to provide those at the infrastructure level.
(Except when you use cases do require them...)
Load balancing and clustering are useful techniques, but they also require expert maintenance and I have seen far more high availability clusters crash catastrophically than I have seen admins break two different servers at the same time.
With regards to NTP, you either provide a single NTP server to your clients, or optimally you configure 4 or more...
Ntpd needs a majority of servers to agree on the time before it can sync.
If you only have one server, that server will be believed. That has the benefit that at least all your systems will be set to the same (albeit maybe an incorrect) time and you can still correlate security events and rely on time dependant protocols such as Kerberos.
If you have two servers behind a loadbalancer your clients behave as if there is only one NTP sever. If time on those two back-end servers differs (depending on your loadbalancing algorithm) either half of your servers will use a completely different date from the others or you might get to see random jumps in time when failover does occur.
If you configure multiple NTP servers most (if not all) NTP client implementations are capable (if you have configured sufficient servers) to distinguish a "false ticker" and disregard it by reaching a majority vote.