1

We recently had an electricity outage of 1-2 seconds at one of our buildings. In this building we have 6 UPS and 4 of them died while/after the electricity outage and all the servers were powered off. Every UPS has only 1 server attached to it. And the UPS are rated at 600W. While the largest server uses max 500W on full load.

I'm in no way an expert with UPS or electricity. But we thought we had it covered with this UPS and also thought that we had some overhead. However this pretty much came to us as a kick in the teeth.

What are possible reasons these 4 UPS got killed?

The UPS we used is APC Smart-UPS 1000VA.

Efekan
  • 171
  • 7
  • Your servers went down because they died AFTER the outage? You don't have the second PSU of the servers connected to a non-UPS power source? – Gerald Schneider Oct 24 '17 at 12:00
  • @GeraldSchneider Our servers only have a single PSU. We had this setup for electricity outages. – Efekan Oct 24 '17 at 12:03
  • 1
    Is it possible you had a particularly bad overvoltage / surge shortly after the power was restored? Is the ground good in the building? In an overvoltage condition, most UPS will attempt to direct to ground. If they cannot do that (such as if the ground is bad), metal oxide varistors within the UPS will absorb the overvoltage, which normally kills the UPS - it will sacrifice itself to protect the connected equipment. – Tim Oct 24 '17 at 12:15
  • @Tim That's definitely possible. If that were the case I'm happy it didn't fry our servers. The location is in a new building for university students. So I would believe that the ground would be good. I will try and check if the ground is good in this building. Thanks for the info. – Efekan Oct 24 '17 at 12:19
  • If your servers don't have redundant PSUs you didn't `had it covered`. An UPS can fail, just like the regular power. If you need it safe you have to use at least one UPS backed power source and one without it. And of course, monitoring for failed UPS devices. – Gerald Schneider Oct 24 '17 at 12:23

3 Answers3

1

It depends on what happened during that outage. I take it you have checked the fuses and the UPSs are truly dead. If there was a strong power spike over the rating of the UPS when the electricity came back that can do it. Or if the feed is not grounded correctly -- especially if you are in an area prone to lightning strikes.

And it sounds like over-doing it, but you should have a UPS to protect your UPS. If the power glitch (most often a spike) is large enough, it can take out almost any protection device. In particular, any retail UPSs that are portable enough to be carried by 1 person.

Mika Wolf
  • 169
  • 3
  • Quick mention: APC Units come with a Lifetime Equipment Protection Policy, so if you do have any damaged kit sitting behind the UPS, you can get it repaired. [US Claims][1] [EU Claims][2] [1]: http://www.apc.com/us/en/faqs/FA156534/ [2]: http://www.apc.com/support/service/equipment_protection_policy_emea.cfm?ISOCountryCode=AM – Dave Oct 24 '17 at 13:08
1

What are possible reasons these 4 UPS got killed? Frequency problems, overload, underload or a cobination of those.

The physics is actually much easier than it seems at first glance: Power generators are engines just like the everyday ones we see all around in our cars, lawnmowers, snowblowers, etc.

Except for new power sources like some wind and solar systems with electronic inverters, the vast majority of power is supplied by large rotating AC generators turning in synch with the frequency of the grid. The frequency of all these generators will be identical and is tied directly to the RPM of the generators themselves, generally 3600 RPM for gas turbines and 1800 RPM for nuclear plants. If there is sufficient power in the generators then the frequency can be maintained at the desired rate (i.e. 50Hz or 60Hz depending on the locale).

The power from the individual generators will lead the grid in phase slightly by an amount roughly corresponding to the power they deliver to the grid.

An increase (=defect) in the power load will cause this rotating frequency to drop.

So a battery systems like this is designed to keep short-term fluctuations in power requirements from dropping the frequency because of lags in the governors and generators which require a finite time to adjust to the new power requirements. These "frequency regulator" power stations can supply very high power for short bursts to keep the power requirements even so that the other generators don't see too much load faster than they can respond due to mechanical limitations.

If this does not happen somehow, your UPS will be killed.

bjoster
  • 4,423
  • 5
  • 22
  • 32
0

I will comment late because this has happened several times to me.

You probably purchased the UPS at the same time and a while ago. The batteries on your UPS were close to failing and the power outage tipped them over. This happened to me a couple of months ago with APC Smart UPS 1500 Tower (4 of 6 failed at the same time). Only the batteries failed, so we purchased new batteries, plus created a schedule to replace batteries every 12 months.

Second situation for you is that your UPS systems are undersized. 600 watts is just too small for servers. We also use APC Smart UPS which are twice as expensive as the lower end models and put a minimum of 1500 VA per server. The servers do not use that much power but the batteries are larger and better quality and can handle longer power outages.

John Hanley
  • 4,287
  • 1
  • 9
  • 20