11

This is something that has had me stumped for over a year. I have 3 APC SmartUPS 3000's and a few non APC units in my server room. Each unit has it's own dedicated connection to the breaker panel.

It seems that I am having to replace the batteries in these units once every 4 months or so. We rarely ever have a power outage, as in not a single outage since the last time I replaced the batteries. I thought maybe it was bad line voltage, so I logged the UPS's output and I did not have a single sag or over-voltage scenario in those 4 months.

Maybe the lines were wired wrong? I tested them and traced them back to the panel and they all checked out.

I decided that the only possibility was that the replacement batteries I was using were not that great (Duracell). I proceeded to buy two brand new APC UPS units with new APC batteries of course, and after four months they were both dead.

The units never register an event in which they switch to battery power, they just simply report bad batteries after 4-6 months. This is a recurring problem that affects multiple units of different models and different brands of batteries. I have no clue whats going on here, I've had electricians and the provider test for various things, and it always comes back clean.

At this point I can only think of something being wrong at the provider level (bad transformer?), but usually that will cause voltage issues. I have also heard of people having phase shifting problems, but I have single phase power, so I am not sure if that is relevant or not.

Note: I used to be an electrical engineer and I have a fairly high level of understanding on how this is supposed to work. So you don't have to dumb down your answers :)

Dave M
  • 4,494
  • 21
  • 30
  • 30
TriadicTech
  • 448
  • 2
  • 5
  • 14
  • 4
    I don't understand why you are asking here without appearing to have spoken to the UPS manufacturer. – user9517 Nov 29 '15 at 23:46
  • 3
    Because the issue occurs across different UPS and battery vendors. It's clearly not related to a single model or manufacture. Other than the two newer units, i don't have active support on the others. – TriadicTech Nov 29 '15 at 23:48
  • 6
    I really don't see how tha makes a difference. Work with the manufacturers rather than having random people on the internet guess because wih the info supplied that's all anyone can do. – user9517 Nov 29 '15 at 23:51
  • 3
    Unless someone else has run into a similar problem... I mean the whole point of this site is for people who share a common career / passion to ask questions and receive feedback from each other. Except for the occasional vendors / manufactures who answer questions as well. – TriadicTech Nov 29 '15 at 23:55
  • 1
    Interesting problem triadic, ignore the 'write to apc' comments since you clearly stated that the issue occurred also with non apc units. The only thing that comes to mind was a series of motherboard failures a small office client had, it was impossible to track down, until one office person noted that the lights dimmed every time a big garbage disposal was run. Turned out the voltage dropped just above the point the APC units would kick in, but just enough to burn out the mobos over time. Removed the garbage disposal, problem vanished. Clearly not your specific issue, but reminds me of that. – Lizardx Nov 30 '15 at 00:00
  • Does the server room have humidity control? Are you near the ocean or other salty environment? – pete Nov 30 '15 at 00:17
  • 3
    Working with one of the UPS manufacturers is actually very good advice here. All UPS units fundamentally work the same way and utilize the same components to do what they do. Whatever is causing the failure is common to all brands because they essentially are all the same. The manufacturer has specific expertise in power engineering, power dynamics, environmental factors related to power and batteries, etc., etc. so working with one of the manufacturers (any one of them) to find the cause of the problem is going to allow you to resolve it for any brand UPS. – joeqwerty Nov 30 '15 at 00:36
  • 2
    I used to be an electrical engineer too, and I'd work with the manufacturers to work out what's going on. I know the limits of my knowledge, and the limits of randoms on the Internet. – womble Nov 30 '15 at 01:01

3 Answers3

10

That is not well known feature of the APC Smart UPSes.

Each time your UPS goes back on the wall and the batteries are recharged, controller counts the energy stored in them. Acid accumulators have some voltage that means "fully charged". When that voltage is reached charging is stopped and new calculated value of the battery's capacity is stored in the controller's NVRAM. Under any circumstances controller do not overhead that value, even if the target voltage is not reached.

When UPS operate native batteries, their capacity is slowly degraded with time. Therefore charging always stopped on the voltage limit, not capacity value. But when you have replaced the batteries, USP knows nothing about that and used an old capacity value to the fresh batteries. That is why they are always charged like they are old and degraded, never reached the needed voltage.

Chemistry of the acid-lead accus require them to be fully charged each time, or, being undercharged regularly, they degraded very, very fast, exactly as you have seen.

APC provide battery replacement service that include reset of controller to the initial values. Sure, in that case controller deal with batteries properly.

There is two news, as usual.

Bad one is that APC's controllers have proprietary protocol and should be resetted by certified APC engineer.

Good one is that some guys have hacked APC's protocol and wrote the tiny tool UPSDiag that allows to connect to the controller and perform the reset. Also you SHOULD use the serial cable, not the USB one.

Unfortunately, APC protocol wasn't hacked completely, so you have to be a bit lucky to log into controller. Once I've try over 20 times until being logged in. Then you have to perform calibration. Capacity value initially is set to some really big number. Then UPS charges batteries until their voltage reach limit. Then UPS drain the batteries to the voltage equivalent to the 25% of capacity. And then charge them again to the full. You have to connect some load to the UPS for calibration, at least 40% of it's power. It is about 600W for Smart-1500, 1.2kW for Smart-3000 and so on.

During that cycle real actual capacity of batteries is calculated and stored in the NVRAM. Voila, you have properly replaced batteries.

By the way, after 5-10 cycles batteries significantly increases their capacity, as well as maximum discharging current. Capacity growth can be about 10-15% of the nominal capacity. So good practice is to recalibrate batteries again after month or two.

Kondybas
  • 6,864
  • 2
  • 19
  • 24
  • The easiest way to log in and reset the APC UPS battery constant (from a Linux host) is to use the `apctest` feature in the [apcupsd utility](http://www.apcupsd.com/manual/manual.html#resetting-the-ups-battery-constant). It's automatically sets up the serial port parameters to accomplish this. – ewwhite Nov 30 '15 at 12:32
  • `apctest` does not perform the reset. It is initiated so-called "self-test" - draining batteries to the 25% and full recharge. That cycle helps to prevent degradation of batteries and can be programmed to be performed periodically. But the only way to reset capacity counter is to use service utility - native or homebrewed. – Kondybas Nov 30 '15 at 16:17
  • [_"The instructions for resetting the battery constant are as follows: Shutdown the apcupsd daemon; Run apctest; Choose option 6 to enter terminal mode;..."_](http://www.apcupsd.com/manual/manual.html#resetting-the-ups-battery-constant) – ewwhite Nov 30 '15 at 16:18
  • This is quite fresh edition, past in the 2007-2008 that procedure hasn't been described in the `apcupsd` manual. – Kondybas Nov 30 '15 at 16:23
  • Sorry, been a while since i checked this. The problem was with the UPS units using old calibrations on the new batteries, killing them almost instantly. We thought apctest was resetting the values, it was not. – TriadicTech Oct 11 '17 at 20:58
6

My thought is that you should have an electrician check the quality of the power. I have this same issue in a number of the plants I support. Our issue is that the plants have dirty and uneven/low voltage due to our server rooms power not being separated from the manufacturing equipment. This causes the ups's to work hard to keep voltage steady and causes the batteries to fail one to two times a year at my worst plants.

Aaron
  • 61
  • 2
  • This is likely the culprit. I have multiple warehouse facilities around the country, and those with wide power variations or frequents dips/sags tend to accelerate battery wear and decrease lifetime. – ewwhite Nov 30 '15 at 00:59
1

I have worked on UPS for over 15 years, for many manufacturers. It sounds to me it my be a problem with what those UPS are supporting. Is it a switching power supply and are the UPS close to being fully loaded?

The reason I ask about a switch power supply is that when the power supplies switch back and forth rapidly it can cause the output of the UPS to be pulled down enough to cause the UPS to pull from the batteries, even though there is not a "utility or outage" issue. This can also be the cause if the UPS is heavily loaded it can cause a similar scenario.

UPS Technician
  • 11
  • 1