APC Smart-UPS: remote power shutoff?

Question

Given:

1. A server room containing about two dozen rackmount servers and a PBX.
2. About a dozen APC Smart-UPS 2200s providing power to this equipment.
3. A fire suppression project requiring immediate shutoff of power to all servers in the event that the fire suppression agent is released to extinguish an electrical fire.
4. We assume that, using separate hardware, we will be able to generate an SNMP trap from the fire suppression system's alarm relay output, and that this will trigger some kind of a UPS power shutdown script on our Nagios server (which fortunately does not share a rack or a UPS with any other servers).

Question:

1. Is it possible to shut down the power to all devices plugged into an APC UPS via SNMP, assuming that we equip each UPS with an ethernet card? The documentation that I've seen refers extensively to the ability to shut down servers gracefully in the event of a power failure via PowerChute; in this case, we would not have time to shut them down gracefully.
2. Does it matter which model of ethernet adapter we use in order to accomplish this? It appears that the old 10Mbps models (AP9606) are available refurbished for quite a lot less money than it would cost to purchase the current model cards; I am unclear as to whether there are significant differences in feature set, other than ethernet bandwidth.
3. Would it be better, for some reason, to use serial cables instead of ethernet cards?

The intent of this question is to tap into your real-world experience with APC UPSs. To date, we have not used any remote management features, and the documentation that I have found to date does not seem to be clear as to whether power can be shut off unceremoniously via SNMP.

Answer 1

Depending on the precise model of UPS, your best bet is to wire the EPO port on the back of those UPSes to the EPO switch on the fire panel.

Smart-UPS 2200 manual, see page 6.

Since these UPSes actually have an actual Emergency Power Off function built in, in order to make your fire-inspectors happy you need to wire those ports to the EPO function of the fire-panel itself. I'm not an electrician, so I don't know the specifics of how you'd wire 10's of these to a single EPO port on the fire panel, but that's a question for your fire panel system vendor.

Half-assing it with suicide software monitors on servers connected to each UPS is the kind of close-enough that gets fire-inspectors to shake their head in that special way that means you'll have to do it all over again, but right this time.

Answer 2

Note that there is a difference between "shut down" and "unpowered" when it comes to computers. When a server shuts itself off, the board remains powered and the server is fully capable of turning itself back on again (think of wake-on-lan). The only way to un-power an ATX-supplied board is to cut the power at (or to) the power supply. In a fire-surpression case, you want all power removed from the server, not just have the server shut itself off.

That said, if you have any advanced notice at all, the command you should send to the server isn't "gracefully shut down" -- that can take minutes to complete. Instead, your goal should be limited to protecting data integrity. An emergency sync and unmount of all filesystems is a much better option in that it usually takes less than a second to complete, which could be all the time you have.

echo s >| /proc/sysrq-trigger
echo u >| /proc/sysrq-trigger

Answer 3

I for one would never sleep soundly knowing that a Nagios / SNMP solution was doing the job of fire supression.

The only choices I would be comfortable with are: something inside the UPS itself triggers a shutdown when it detects overheating (e.g. ambient temperature significantly above normal); or hard wiring to the EPO port (sometimes called REPO port) with some sort of thermal relay or similar. You want this to work, without the slightest shadow of doubt, under all situations.

Answer 4

I've worked with the APC powerchute software and APC management cards quite a bit but I'm struggling to see how you can achieve this with your current APC's. You've got two scenarios to contend with:

1. "Normal" power loss where you want the UPS to provide power to the connected equipment and initiate a graceful shutdown of said equipment.

2. "Abnormal" power loss initiated by the triggering of the fire supression system where you want the UPS to remove power from the connected equipment immediately.

My question is: How will the UPS differentiate between the two events? How will the UPS detect the second event? You definitely need something more upscale than your typical UPS that can tie in to and monitor the fire supression system. A Smart-UPS 2200 just doesn't have this capability (that I'm aware of).

Answer 5

I have used NUT (Network UPS Tool) on Linux to monitor and control APC UPSs over serial ports, USB, and SNMP. (You may need some multiport serial cards to handle a bunch of UPSs.) There are clients for Windows, and the software runs on most Unix flavors.

NUT has options to force a shutdown before the batteries run down. Depending on the command set power can be shutdown immediately, or a controlled shutdown can be initiated immediately. Intersystem control is done over the systems normal ethernet port.

The remaining problem would be to inform the master controller that fire suppression has been triggered.