63

Suppose I have a rack with several servers and other stuff. One of servers overheats severely and either starts smoking or catches fire while there's a serviceman nearby.

If anything similar happens in an apartment and there's a fire extinguisher nearby using the latter promptly often lets extinguish the fire very fast but in case of server rack improper extinguishing may lead to unneeded extra damage to surrounding equipment.

To clarify, I'm talking about a really small fire that one can try to extinguish without risking their life - like grab a nearby extinguisher, discharge it and get the fire extinguished in say ten fifteen seconds.

What's the strategy to extinguish a small local fire in a server rack? What type of extinguisher is to be used? How to minimize damage to surrounding equipment?

dunxd
  • 9,482
  • 21
  • 80
  • 117
sharptooth
  • 2,727
  • 4
  • 32
  • 38
  • 53
    1) Hit the fire alarm button in your datacentre. 2) Run. – Tom O'Connor Oct 25 '11 at 09:09
  • 1
    @Tom O'Connor: That only minimizes risk to one's life. – sharptooth Oct 25 '11 at 09:25
  • 4
    This really should be addressed to your local fire prevention office. – user9517 Oct 25 '11 at 09:34
  • 19
    @sharptooth yeah, and? A company's hardware is insured, and easily replaced. You're a lot harder to replace, as your significant others / dependants will agree. – Tom O'Connor Oct 25 '11 at 09:35
  • @Tom O'Connor: Well, I know that, still running away when I'm sure I can spend fifteen seconds to at least try seems quite stupid. – sharptooth Oct 25 '11 at 09:39
  • 6
    Not so much. A halon fire suppression activation isn't a fun time, and plastic can burn with poisonous gasses. Either way i wouldn't want to hang around. Opening the server up and providing more air to burn isnt going to help a whole lot. So it depends, if its super small fire and you have an extinguisher near by and can yank the server out and spray it with something electricity safe then maybe, but i wouldn't risk it for anything even remotely dangerous. even burnt fingers suck when you're footing the bills. – Sirex Oct 25 '11 at 10:24
  • There's a great phrase on most fire policies (ie, company rules) "If you feel capable of tackling it, do so, but do not take any risks" – Tom O'Connor Oct 25 '11 at 11:10
  • 8
    This question doesn't sound hypothetical to me. If you're actually trying to come up with a way to justify keeping an actual, known fire risk in your data center, you need to stop and really think if it is worth the risk. – RobW Oct 25 '11 at 17:05
  • 1
    The biggest problem with this "localized fire suppression tactics" myth is that the first step to extinguishing an electrical fire is to TAKE AWAY THE ELECTRICITY. If you don't have an EPO switch hooked up to every one of your UPSs, you can pretty much forget about it... – Skyhawk Oct 25 '11 at 18:37
  • 3
    Minimising the risk to your life is worth more than any stupid servers sitting ina data center. If you're comfortable in the knowledge that you have off site backups, *get the hell out of there*. If you want to quantify it, the cost to your insurance to replace the entire building and all its contents is probably a few million. The cost to your insurance (and public liability) if you die would be subtantially more, not counting the fact that you're dead, which your family will then have to deal with. – Mark Henderson Oct 25 '11 at 19:29
  • 3
    **That only minimizes risk to one's life** - you say it as if there's something wrong with that. There isn't. – Rob Moir Oct 26 '11 at 08:43
  • 1
    Are you a manager, perchance? – oracle certified professional Oct 26 '11 at 11:32
  • @oracle certified professional: Nope, I'm a software developer. – sharptooth Oct 26 '11 at 11:51
  • 2
    Leave the fire suppression to the professionals and the server operators then. Smoke from a server fire is physical, lethal and not covered by a software algorithm. Your best fire suppression is to cut the power to the rack and run. – Fiasco Labs Jan 13 '13 at 05:46

10 Answers10

79

This is a strange question, but I'm gonna attempt to answer it anyway.

All electrical fires must be extinguished carefully. Especially if they're still live. Fire departments will all recommend the use of a CO2 extinguisher.

In a datacentre environment, however, I'd (But I don't recommend you) do one of two things.

  1. Hit the EPO (Emergency Power Off) and the fire alarm button.
  2. Find a CO2 or Halon* extinguisher and attempt to extinguish the fire
  3. If that doesn't work. Goto 1.
  4. Unconditional Jump: Run out of Fire Exit.

I'd be more concerned about the protection of the rest of the suite than the expense or damage caused by following their procedure for fires in the building. You're operating as a business, so you've got insurance, both for your hardware, and Public Liability. It'll all be covered by that if you've got a decent insurer.

It'd be better all around to hit the fire alarm and let the professionals deal with it. Electrical fires produce evil acrid choking smoke, often containing horrible toxins, and I sure as hell wouldn't wanna be trying to extinguish it without breathing apparatus.

*Halon is a bit of an odd one. It's a fantastically efficient extinguishing agent, but has a nasty side-effect of destroying the Ozone layer. I've only seen Halon extinguishers in datacentres, and even then, there's a lot of paperwork if you discharge one. It works by depleting the oxygen in the applied area, and as a result, it'll suffocate the fire, and you at the same time if you linger too long. Again, this kind of lends itself to calling the fire service. Or activating the building's fire supression system. Don't worry about the expense, that's what insurance is for.

Also.. has this just happened? I can't resist this.

Leave building before tweeting

(Disclaimer: I'm an ex fire-cadet, so I've got a pretty good knowledge of fire and it's extinguishing. )

Tom O'Connor
  • 27,440
  • 10
  • 72
  • 148
  • Shouldn't item 3 read "... goto exit"? – sharptooth Oct 25 '11 at 09:26
  • 9
    +1, totally agreed. Electrical fires, and fires in general, are very dangerous. Hardware can be replaced. Software can be reinstalled. Data can be restored (you have backups, right?). Lives can't be revived. – Matthew Scharley Oct 25 '11 at 09:28
  • 19
    I'm a developer, that's why I suggested that you change item 3 to "in that doesn't work - run", item 4 in your "algorithm" will not save one's life in case of danger - they'll be looping between items 1 and 3. – sharptooth Oct 25 '11 at 09:41
  • +1 for CO2 extinguisher. Using foam or chemical will surely destroy your equipment for good. – pauska Oct 25 '11 at 09:57
  • @pauska and possibly you, in the process. – Tom O'Connor Oct 25 '11 at 10:13
  • 23
    You might want to test that pseudo-code for potential fatal errors... – e100 Oct 25 '11 at 14:56
  • 14
    You guys do realise that you're not computer programs, right? You're capable of exiting a for/while or a do loop without specific instructions to do so... – Mark Henderson Oct 25 '11 at 19:32
  • 2
    @sharptooth goto considered harmful. And exit() is TERRIBLE for error handling. – fluffy Oct 25 '11 at 20:08
  • @fluffy: Who said `goto`? `break` would do. – sharptooth Oct 26 '11 at 07:12
  • 1
    Of course Halon is not used any more - instead we have halon replacements which are better for the ozone layer, but can kill you faster (some not only displace oxygen, which is quite difficult for humans, but can react with the moisture in your mouth/nose to produce nasty toxic substances) – Rory Alsop Oct 26 '11 at 12:09
  • Halon isn't used anymore in the EU.. Some places still use it.. The US military still loves the stuff (According to Wikipedia) – Tom O'Connor Oct 26 '11 at 12:19
  • The "halon" (not sure what's in the tanks) systems in the com-rooms where I work supposedly don't release enough of whatever they use to be a suffocation hazard; making frostbite if you stand under the nozzles the biggest safety hazard of the system. Caveat: I'm just repeating what I was told in the annual training sessions and haven't actually hit the big red button and stayed inside to test the claims. – Dan Is Fiddling By Firelight Oct 26 '11 at 12:55
  • @sharptooth You did, for one. – fluffy Oct 26 '11 at 17:22
  • @DanNeely - if you're standing underneath one, frostbite around the face is still a serious hazard. Don't forget that. And there have been reports of pressurised fire suppression systems that discharge gas fast enough to break windows and even rupture (thin) walls. The modern gasses might be better than halon but I'm still not sure I'd care to be standing near one of the vents when it discharged. – Rob Moir Nov 01 '11 at 22:36
  • @RoryAlsop with the exception of Aerospace where it's kept around for suppressing avionics/aircraft electrical fires. Despite the cessation of production in 1994, there's still enough around to keep us supplied. H3R supplies these for aircraft use. – Fiasco Labs Jan 13 '13 at 06:09
  • @Fiasco - interesting. I didn't realise anyone still used it :-) – Rory Alsop Jan 13 '13 at 11:47
24

Speaking from organizational experience at my last job, a small fire in one rack can turn into a big fire with a whole rack, and very shortly afterwards, the rack next to it as well. Fire spreads fast. Once the insulation jackets of all of those power and Ethernet cables gets involved it travels. It travels really fast. Then the flames start tickling the ceiling tiles and things get really bad.

We know this from experience, though I no longer have access to the photos since I don't work there anymore.

When you get a fire, any fire, you need to treat it like you would a larger fire. Since server-rack fires can burn fast and hot, you need to have enough fire suppression in place to handle something like that:

What's the best fire suppression for a server room?

Our experience with this issue was related to having insufficient suppression in the area the servers were kept. There may have been sprinklers in the room, but considering the age of the building that was unlikely. There were CO2 extinguishers, but the problem hit when no one was in the building and therefore present to use them. By the time the Fire Department got on site, the building was well on its way to gutted. Bother.

The server room that was involved in this conflagration had maybe one point of the items on that 'best fire suppression' answer. Don't be us.

Community
  • 1
sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
  • 3
    And PVC smoke from all those wire jackets is very quickly lethal to human beings. And PVC quickly melts and turns into a very efficient drip torch system helping the spread. So, yank the nearest alarm and run, or become a heroic statistic... – Fiasco Labs Jan 13 '13 at 05:57
24

Definitely extinguish the fire without regard for the rest of the servers on the same rack. Any properly designed data center will have backup servers at another location for the applications to fall back on. It is better to lose a rack of servers than to lose a life or the whole data center.

MartyTPS
  • 256
  • 1
  • 2
  • 2
    Agreed ... not extinguishing the fire means it damages the whole room, so if you only hurt one rack, or the racks immediately next to it, you're still better off. – Joe H. Oct 25 '11 at 18:19
  • Cut power to the rack before bothering with a fire extinguisher, the heat source needs to be eliminated before the fire extinguisher will be useful in suppressing the fire. And A/B class extinguishers are as useful as putting water on a magnesium fire. – Fiasco Labs Jan 13 '13 at 05:48
  • This is a bit old, but it was part of the question that I don't see answered: Energized electrical equipment fires should be extinguished with a class C fire extinguisher, but the FIRST thing you should do is cut power (at the breaker/UPS. Please don't try to turn off the power supply on a burning server :p ) – Daniel B. May 24 '13 at 05:29
9

If you don't know what to do in the event of a server room fire then you shouldn't be in the server room...

By which I mean that if you have enough computers and switch gear in one place to call it a "server room" then you have enough equipment in there that could be a potential fire risk, and at that point someone needs to have a sit down and have a think about procedures for dealing with a fire in that room and all the staff who are allowed in the room should then be trained in that procedure.

You should at the very least have a CO2/Halon/similar hand-held extinguisher in that room and training available for all people who might have to use it.

If the room is large enough and/or contains enough valuable equipment then an integrated fire suppression system should be considered. We have them in both of our server rooms, with FM-200 being our gas of choice as its a fairly new install, rather than Halon, which I don't think you're supposed to use in new installs these days, at least not in the UK.

In both cases, while I'm not a health and safety bore nor do I wish to pretend to be one, this is one area where the right steps to take in the event of a fire in the server room should be given to someone before they are allowed to work in the server room, and the first step in the list of procedures should include a warning about risk of death/injury and instructions to stop fire-fighting attempts and evacuate if there are any possible doubts.

Rob Moir
  • 31,664
  • 6
  • 58
  • 86
6

We have a CO2 extinguisher in our comms room. I guess I'd use that if I was in the room when the fire started, and noticed it. Couple of discharges at the base of the fire, then get out even if it appeared to be out, then raise the alarm.

The only time we had a fire in our comms room, it was the AC unit that started it, and luckily it didn't spread. No one stayed around to tackle it - the smoke detectors caught it before anyone did, and the building was evacuated. Having the previous nights backup tapes in a pelican case next to his desk, ready for collection by our offsite storage suppler, our network manager carried those out with him.

The fire brigade came, and it was valuable to let them know the fire was in the server room, and give them an access card. They didn't need to take an axe to the door or destroy anything - they went in with breathing apparatus, located the fire and dealt with it. So one thing worth doing is being around to talk to the fire brigade when they arrive, rather than disappearing down to the pub with everyone else...

It would have been a major disaster for our organisation if the fire brigade had run in and sprayed everything with water. Yes, we have an offsite recovery where we believe we can recover our systems within 48 hours. But we would really rather avoid having to use it.

dunxd
  • 9,482
  • 21
  • 80
  • 117
  • 6
    "Yes, we have an offsite recovery where we believe we can recover our systems within 48 hours." - You should test this annually, rather than trust that it will work in an emergency when you need it. – mfinni Oct 25 '11 at 14:23
  • 1
    You really shouldn't need to give the fire department an access card in the case of an emergency; your physical security systems should get overridden during the emergency. – RoundTower Oct 25 '11 at 22:19
  • @mfinn - yes, we test the site annually. Still don't want to use it though. – dunxd Oct 26 '11 at 11:43
  • @RoundTower - in an ideal world that would be in place. Welcome to my real world... – dunxd Oct 26 '11 at 11:43
6

Halon gas has been banned in most places, for many years now.

The biggest problem with the hand held fire extinguisher is that it is unlikely to be near the fire, unless it is in a very small server room. By the time you go and get it and then get back to the fire it will be too late.

Also, I took an introductory course in firefighter training... one of the first things they taught us is that those portable extinguishers don't hold very much. It is an often fatal mistake to be overconfident in the size of the fire that you can put out with one. Basically, anything bigger than a small trash bin is beyond their capability. Also If the extinguisher is not specifically rated for Electrical Fires then do not use it, you can actually make things worse and potentially electrocute yourself too.

As an electronics technician, with considerable experience in ~smoke~ testing components, ;-) I have found that killing the power is the most effective way to put out a small fire. Most electrical fires are self extinguishing once the power is removed.

So the best thing is to kill the power and leave. Given how toxic the fumes can be, either stay low or hold your breath as you sprint for the door. Don't try to be heroic, you can get pretty bad lung damage in a very short time. If anyone else is in the room do let them know too.

Any largish data center will have automatic detection and protection equipment. if your's doesn't then it is time to talk to the higher ups. Also many fire departments have excellent educational resources available, if you are hitting resistance from management about this, the fire department's outreach program may be able to help you make the case.

codeslinger
  • 161
  • 1
  • 4
4

Naively, I would expect that any carbon dioxide or nitrogen (basically inert gas) solution should be good. It looks like there are commercial solutions out there that are even human-breathable.

adavid
  • 140
  • 3
  • 6
    by human breathable, they mean - you can survive for the time it takes to get to the exit. They work by suppressing oxygen, and us humans need a bit of the old O2 – Rory Alsop Oct 26 '11 at 12:11
  • Oh sure. But at least it is not (too) corrosive to mucosa ;) – adavid Oct 26 '11 at 14:40
  • Halon 1301 can suppress fire without displacing all the oxygen and without being immediately deadly to humans (though it's still advised to get out if you can). Last I heard it was still be used by Eurotunnel on their car shuttle trains for this reason. – Peter Green Nov 27 '19 at 12:32
3

There should be no inappropriate fire extinguisher near the server room and the correct extinguisher should be readily available and clearly labeled. A fire blanket should also be available and in the situation being asked about that's what I would use if it's practical to do so. Either way, try not to breathe the fumes, which will be quite toxic.

If you can't extinguish the fire in a matter of seconds then sound the alarm, get out of there and leave it to the professionals.

John Gardeniers
  • 27,262
  • 12
  • 53
  • 108
2

The other option is a fire suppression kit designed for racks, if they detect a fire they'll flood the room with an innert substance to stem the fire before it takes hold. It's pretty expensive stuff but if it's a business necessity it's the best way to do it.

I'm no expert on them but take a look at http://www.cetecglobal.com/technologies/security/fire_suppression.htm as an example.

Alex Berry
  • 2,307
  • 13
  • 23
2

At high end data centers, they will typically have a fire suppression system.
This systems will release a gas agent that will inhibit the chemical reaction that is fire.

Typically, this means the air is no way close to breathable, and so an alarm will air and the whole area will need to be evacuated.

http://en.wikipedia.org/wiki/Gaseous_fire_suppression

For reference, Halon 1301 is what I've seen used.

user606723
  • 544
  • 1
  • 4
  • 10