Maxing out both PDUs in a rack with redundant power

Question

If I have a rack full of servers using redundant power supplies, and I'm keeping the amps at 80% (16A of 20A), isn't this still asking for trouble in the event of a power loss on one of the circuits?

If power is distributed 50-50 (e.g. an average of 1A per power supply on a server that would use 2A when using only one power supply), when one circuit fails, the load is going to pull from the other circuit (and will likely throw a breaker).

I guess I've always just gone the 80% route without thinking too much about it, but does this mean I should only be "maxing out" our circuits to 40%?

Random idea: Perhaps you could have 3 power supplies to the rack, then have 1/3rd of your servers connect to each combination of 2 from the 3. In the event of 1 failure, you only add 50% more load to the other circuits, not 100%. So to run 20A of servers, you need 30A of power, not 40A. — rjmunro, Oct 10 '13 at 23:47

score 8 · Accepted Answer · answered Oct 03 '13 at 17:11

Running dual-PSU servers in a balanced power distribution mode doesn't mean that they each use exactly 50% of the power that a single PSU would use. It's usually horribly inefficient to balance the load. I don't have the exact numbers, but afaik it's about a 20% waste in power.

However - running them in balanced mode and then push each PDU up to 80% is begging for trouble. I see absolutely no chance in the remaining PDU surviving if you're already that far up in usage with balanced power distribution.

One other thing to keep in mind is the most catastrophic scenario in this situation - a site-wide power failure where both PDU's goes offline, and one of them breaks it's circuit because of overvoltage or similar (not that uncommon). What happens when all your servers power on at the same time, with a single 20A circuit to share? I'm sure I don't have to explain it further..

If I were you I'd start making preemptive changes right away.. set the servers to high efficiency power (if they support it), which usually gives you a 2-3% usage on the standby PSU and 100% load on the other. You'll quickly see if you're maxing out one of the PDU's. Also - make sure that the servers have power-on delayed in clusters to make sure that they all don't power on at the same time in case of outage.

Edit: Oh, and of course - order bigger PDU's and bigger circuits :-)

I ordered a 30A circuit on one cabinet (this is a 5 cab down to 2 cab consolidation effort) because I was close to 80% on both PDUs in the target cabinet. Sounds like we should be going 30A on both cabs and dropping MORE servers. Ugh. Power sucks. — Matt Beckman, Oct 03 '13 at 17:59

ewwhite · Answer 2 · 2013-10-04T18:35:25.590

Much of this depends on your tolerance for risk and the policies at your datacenter facility. If this is your office's datacenter/computer room, the conditions may be different than at a co-location facility.

My colo strictly enforces the 80% utilization rule. This is for their protection, my protection and customer/provider SLAs. But it's also mandated by the electrical code here.

High-end server power supplies can be configured to balance load across the circuits or to run in a high-efficiency mode, where the load is unbalanced...

balanced PSU mode on an HP ProLiant server

hpasmcli> SHOW POWERSUPPLY 
Power supply #1
        Present  : Yes
        Redundant: Yes
        Condition: Ok
        Hotplug  : Supported
        Power    : 105 Watts
Power supply #2
        Present  : Yes
        Redundant: Yes
        Condition: Ok
        Hotplug  : Supported
        Power    : 95 Watts

hpasmcli> SHOW POWERMETER 
Power Meter #1
        Power Reading  : 200

high-efficiency mode on an HP ProLiant server

hpasmcli> SHOW POWERMETER 
Power Meter #1
        Power Reading  : 290
hpasmcli> SHOW POWERSUPPLY 
Power supply #1
        Present  : Yes
        Redundant: Yes
        Condition: Ok
        Hotplug  : Supported
        Power    : 255 Watts
Power supply #2
        Present  : Yes
        Redundant: Yes
        Condition: Ok
        Hotplug  : Supported
        Power    : 35 Watts

Your situation sounds like mine. You have two 20A feeds and are using redundant power supplies. You really don't want to load either side more than 40%. You want the aggregate to be 80%, so that's 16 Amps for you. No more!

As you can see below, I run a bit close to the edge. I had a temporary server in the environment for 45 days and had to receive special permission from the facility to run over the power limit for the duration.

enter image description here

score 2 · Answer 3 · answered Oct 03 '13 at 17:11

In my experience, the server power supplies do not behave as you predict. In other words, a server drawing 1/1 in a steady state with both PDUs working will go to 1.5/0 or 0/1.5 if a PDU or circuit dies. You can test this for yourself with a smart PDU or with a kill-a-watt.

The real question is whether the 80% load rule is an absolute requirement or not.

In my case, I simply stored an extra PDU in the cage and monitored the power closely. Occasionally, we'd drift above 80% utilization on one circuit or another. It never got to the point where the facility complained.

score 0 · Answer 4 · answered Mar 20 '14 at 22:30

Best practice to load each side up to 40% so that in the event one feed/circuit fails, a fail-over to the other feed/circuit can be successful. 80% rule is pertinent and is part of National Electric Code.

If you have 30A circuits coming into your racks, you will see circuit breakers on the rack PDU's themselves. Try to plug your servers in a balanced way across the 2 breakers in a 30A single phase feed, or across the 3 breakers in a 30A 3 phase feed to minimize overloading of any one of the branch circuits.

Maxing out both PDUs in a rack with redundant power

4 Answers4

Linked