Fires in modern data centers are a harsh reality. Such fires result, not only in serious amounts of direct damage, but also in costly indirect damage, including downtime associated with server outages and cleanup from the fire. These fires are not only dangerous, they can disrupt modern life.
In 2017, a fire at a large data center in Azerbaijan took almost seven hours to extinguish. During that time, the entire country lost computer access to the Internet, with access to the network available only from mobile devices[1]. There is also evidence that a fire at a data center in South Korea’s Gwacheon city may have caused Samsung devices (mainly “smart TVs”) to malfunction in 2014[2].
A fire that took place in 2013 at the Macomb County data center in Michigan, U.S.A., forced the local government to declare a state of emergency[3]. Access to municipal government resources, public service databases, even telephone communications were disconnected for several days; and a fire in the Shaw Communications Facility data center in Calgary, Canada in 2012 cut off many government services and delayed hundreds of planned operations at local hospitals[4].
Fires in data centers are not just problematic for the operating company and its customers but can have a broader effect on other regions or different continents.
To avoid severe consequences, it is important to prepare for a fire by properly designing, building and operating data centers, as well as equipping them with modern fire protection systems. In order to minimize risks, fire protection of data centers should be designed considering realistic emergency scenarios, not only fire code requirements.
Specialists in the theory of burning and extinguishing fires (there is such a specialty) have a saying, “Any fire can be extinguished with one spoon of water – if you know where, and most importantly WHEN – to pour it.” An uncontrolled fire at a data center often starts with a small emergency, like a short circuit, the heating of a coupler, or a cable line overload. The temperature rises, the insulation overheats, the pyrolysis of plastic or rubber begins, and then smoke appears, followed by smoldering. This process progresses slowly and at its start, is small in scale. It is a single point in the hall, in the cable channel, in the transformer or in the switchboard. If intervention does not occur early on, smoldering can grow into open flames, and the fire can intensify, quickly spreading through overheated insulation on the boards of computing devices and thermal insulation inside air ducts.
Uncontrolled propagation is what mainly distinguishes these fires from any other combustion processes. Flames spread by heating and igniting surrounding combustible materials. Therefore, the larger the fire, the more heat it emits, and the faster it grows.
It is obvious that it is easiest to manage a fire in the first minutes of its development, when the area of the fire, its heat generation rate and propagation speed are minimal. Textbooks on fire tactics pay considerable attention to the speed of reaction and deployment of fire departments. The speed of deployment and fire reconnaissance allow fire fighters to intercept a fire when it is the easiest to handle. Citizens and officials are also trained to call the fire department when a fire is detected. Intervention of professionals within the first 10 to 15 minutes of the fire starting are what can prevent it from developing to catastrophic proportions.
Theoretically, the optimal time for intervention is the period before the actual start of combustion. If you knew where the cable would ignite in the next minute, it would be easier to prevent a fire – turn off the server or board you need, disconnect the cable line, pour the proverbial “spoon of water” – and ideally, prevent the fire.
Modern technology now allows you to do this. The use of highly sensitive addressable fire alarm systems, in particular, the aspiration type, allows you to turn on an alarm (with an exact indication of the location) even before the appearance of visible smoke, at the time when release of thermal decomposition products of the insulation of overheated cables begins. If the data center personnel quickly and clearly respond to the signal, the beginning of the smoldering process involving an overheated cable would be visible, and this could give you time to take necessary measures and prevent an actual fire from forming.
By starting the fire-fighting process through an automated system, we fulfill regulatory requirements and suppress the fire right when it starts, however, it is not usually that simple. Fire detection technology specialists understand that the sensitivity of fire sensors are directly related to the reliability of fire detection. The more sensitive the sensor is, the earlier it detects a fire, but the greater the likelihood of a false alarm. Dust, chemical vapors in the air and exposure to electromagnetic fields can all cause false alarms, and you cannot fundamentally get rid of the risks of false alarms. Special means to combat false alarms (both hardware and software) exist in fire alarm systems, and are successfully used, but they tend to lead to a «rougher» sensitivity of the sensors, which can slow down fire detection. Either we catch the fire at a very early stage at the risk of receiving false signals, or we detect the fire reliably but at later stages of its development.
If people react to a fire signal, false alarms are not as problematic. Quickly discovering exactly what is happening in a switchboard or server cabinet is typically straightforward for a well-trained, disciplined employee. Automatic fire extinguishing systems can be more complicated.
Traditionally, data centers are protected by gas fire suppression systems. Modern gas compounds are effective agents that are safe for the delicate equipment in data centers. However, gas fire suppression systems can be costly, and the price of a gas agent itself can be a large part of the total. Once a “fire!” signal is received by a fire alarm system, the gas fire suppression system does not immediately discharge but provides a delayed response so people have time to evacuate the protected area. In the event of a fire alarm, data center personnel evacuate the protected premises first and then ensure the doors are tightly closed after exiting (this is a prerequisite for gas installations, so the gas does not escape. Personnel then call the fire brigade, who wear insulating gas masks in the burning room.
Accordingly, the facility owner will pay the cost for recharging the gas fire suppression system, since the gas released in automatic mode cannot be collected again in cylinders. Only after the system is discharged is it possible to discern whether it was a false alarm or a real fire. From a maintenance engineering point of view, the most acceptable solution is to operate gas fire suppression systems only in manual mode and provide gas supply on the operator’s command, but this is directly prohibited by fire safety regulations.
It would seem that if a highly sensitive fire alarm system informed us of a fire, we would have a few precious minutes to arrive at the alarm site, find out what is happening, turn off the server, disconnect the cable, and put out the fire with a fire extinguisher (or make sure that the alarm is false). However, personnel must be quickly evacuated from the premises. The advantages of modern, ultrafast, targeted fire detection cannot be fully exploited using gas fire suppression systems, and other solutions are possible.
Water mist fire suppression systems use, as the name suggests, one of the most effective fire suppression methods – clean water. The effectiveness of water is fully manifested when spraying it as a mist from sprinklers and saturating the fire with water vapor. This effectively cools the combustion zone, precipitates smoke and protects the surrounding equipment and building structures from heating.
Fine water molecules hanging in the air around the fire zone absorb heat radiation, preventing the fire from spreading. Water damage is minimal because water mist accumulates slowly on surfaces. At the same time, pure water has a distinct advantage – it is completely safe for people. This means that decision on evacuation needs to be made based on how fire develops at the scene of the emergency.
But what about false alarms? Spraying water, albeit in the form of a thin fog, onto running servers when there may be no fire at all, does not seem like a safe solution.
That is why pre-action section valves were developed and continue to be successfully used in water mist systems to this day, allowing dual control of operation. In this case, the water mist system pipeline network is divided into a water-filled part, from the pumping unit to the section valve, and a dry pipe part, from the section valve to sprinklers located in the protected room.
Sprinklers in such installations are normally closed sprinklers equipped with a thermo-sensitive bulb that bursts when a certain temperature is reached. The dry pipe part in standby mode is filled with pressurized air. The sprinklers closed by the bulbs do not allow air pressure to fall, while the closed section valve holds water, preventing it from entering the pipeline section located in the protected room.
Upon receipt of the “fire!” signal from the fire alarm installation, the section valve opens the water supply to the pipeline. However, water is not sprayed through the sprinklers if their heat-sensitive bulbs have not burst. Only when the temperature at the sprinklers reaches the set value do the sprinklers open and begin to extinguish the fire.
Thus, two separate events are necessary for the water mist to discharge: a signal from the fire alarm system and the heating of a sprinkler thermo-sensitive bulb.
This technology will not react to false alarms, allowing the fire alarm system to warn of a fire at an early stage, while only extinguishing flames once the fire really starts to develop. If, for some reason, the sprinkler bulb is broken (for example, during repair work in the protected room), a drop in air pressure in the pipeline signals there is a broken bulb and water will not be discharged, since there is no signal from a fire detector.
However, it is still most effective to extinguish a fire in the first few minutes, the moment it ignites, when the damage is still zero and minimal actions are enough to extinguish it.
Can we wait for the sprinkler bulb to warm up to start fire suppression?
This is where employee safety, as it relates to water mist suppression, becomes important.
Water mist systems do not require evacuation of people before discharge, which allows employees to work out the full cycle of actions— arriving at the scene, inspecting equipment and analyzing the situation, disconnecting cable lines or server racks (or individual servers in the rack), switching on backup equipment, and using fire extinguishers to intervene at the right time and with accuracy, preventing the development of the accident into a full-blown fire.
If personnel actions are unsuccessful, combustion can get out of control and begin to spread. At this stage, sprinklers will discharge the jets of fine water mist right into the ignition zone, without impacting the rest of the equipment inside the machine hall. Thus, the water mist system acts as the “last line of defense”, stopping the fire when people were unable to deal with it quickly and effectively, and keeping the situation under control until the fire brigade arrives. Reducing the temperature and smoke precipitation with water mist
discharge not only protects equipment and structures of the data center but also provides safe conditions for the evacuation of personnel and improves working conditions for firefighters.
It should also be noted that water mist systems, unlike gas fire suppression systems, do not require high levels of enclosure integrity. Whether the door to the hall will be closed after evacuation or will remain open is irrelevant. The water mist will work equally efficiently either way. This also reduces the build cost, making water mist a more attractive financial solution.
Ultimately, it is possible to develop a fire protection system that will optimally combine fire detection and automatic fire extinguishing subsystems with organizational measures that allow trained personnel to act effectively to minimize possible damage both in the event of a real fire and in case of false alarms.
In practice, it is possible to provide:
• The use of highly sensitive fire alarm sensors that detect fire at an early stage.
• Safe working conditions for personnel during an emergency fire response and in the first minutes of development, when it is most effective.
• Protection of data center equipment from false alarms, reducing the costs of false alarms to zero.
• Effective control and suppression of a fire, if it was not possible to stop the fire manually.
The combination of qualified, trained personnel equipped with the necessary tools and manual fire extinguishing equipment, highly sensitive air sampling fire alarm systems, and reliable water mist fire suppression systems minimizes the risks of damage and malfunctions under various emergency scenarios. While this combination may not always be an optimal solution, it is vital for a fire safety specialist to have a wide range of technologies in their arsenal so they can analyze real-world risk scenarios.
In fire safety and response, it may be helpful to think outside the scope of traditional solutions, not be afraid to analyze and prepare for emergency scenarios and invest in solutions that will lead to positive outcomes.
[1] URL: http://www.datacenterdynamics.com/critical-environment/data-center-fire-kills-internet-in-azerbaijan/95227.article [2] URL: http://www.cbsnews.com/news/report-samsung-fire-caused-outages-on-devices-smart-tvs/
[3] URL: http://www.datacenterknowledge.com/archives/2013/04/19/michigan-county-offline-after-data-center-fire/
[4] URL: https://www.datacenterknowledge.com/archives/2012/07/16/data-center-fire-disrupts-calgary