New technologies like Docker, Mesos, Kubernetes etc. allow much better server utilisation in an organisation. However, I'd like to know how utilisation can be maximised across two datacenters, taking into account the failure of an entire datacenter.
Given this scenario:
Two datacenters (DC-A and DC-B) with an equal amount of compute resources. Both datacenters are running and serving requests in a load balanced/round robin fashion. The server utilisation in both datacenters is 80%.
Let's say DC-B fails (physical or network) and is unreachable. DC-A will not be able to add an additional 80% utilisation as it's already at 80% itself. Leaving the organisation in state where potentially DC-A will not be able to handle the extra demand and will cause disruptions...
Does this mean that a two datacenter (DC-A and DC-B) organisation can only have 50% maximum utilisation per datacenter? I.e. Either DC fails and the other DC will be able to pick up the slack that the other was carrying (50% + 50%).
Is this thinking correct? How are others handling this problem or am I missing something?