Behind the scenes of the night our transformer shut down in our data center
One night in September, a power transformer shut down in one of our Parisian data centers. Read on to find out what happened during this tense night.
In light of recent events, many of you have asked us about the practices and methods we use to protect our data centers. We thank you for all your questions, and we will respond, as always, with maximum transparency.
We have always considered our data centers to be a selling point and a tangible reflection of the quality of our products. They are also an invaluable production tool, and a source of pride for us.
For almost 15 years, our data centers have been part of an uncompromising investment strategy, which allows us to take full responsibility on our clients’ behalf. We have chosen to specialize in all aspects of this profession: from design to construction and implementation, basing our engineering on feedback. We have used our teams’ valuable knowledge to set up data centers which are more and more resilient and innovative.
We are one of the few companies with a fully integrated approach, from data center to software (as well as networks and hardware), all in full transparency. We are one of the last remaining cloud companies to master the highly specific area of data centers – almost all other companies in the market use large real estate actors’ sites for their international expansions.
We are the only triple play-type cloud supplier to provide all three services: data center and private infrastructure colocation, dedicated high-end servers for maximum control and impact and a modern and flexible public cloud ecosystem.
Our command of the whole value chain allows us to offer competitive pricing and innovative services, while never compromising due to economic considerations.
With regard to data center colocation, we are one of the biggest French players and an important European player. For many years we have been hosting Gartner Magic Quadrant companies on our infrastructure, as well as many other companies which are well-known in Europe, thus demonstrating our stringency with regard to colocation.
The regulatory context: Regulation ensures that people and the environment are protected. The operator, along with the insurer, handles asset protection.
Data centers in France are governed by labor law, and above a certain power, by ICPE-type decrees (establishments classified for environmental protection).
Regulation on this is light-handed and mainly concerns safe evacuation of staff (emergency exits, smoke extraction, etc.) in case of incidents, and environmental protection. For example, it does not require the installation of fire detectors or fire suppression systems, nor any asset-protection measures.
It is important to understand that the technical design and inclusion of asset protection measures in a data center is therefore entirely dependent on the project owner and the operator, but also on the insurer’s conditions on the level of coverage and deductibles. To summarize: these regulations ensure the protection of people and the environment, the operator and the insurer ensure the protection of physical assets.
There has long been confusion between design resilience and certification such as ISO standards. The aim of certification is to standardize governmental practices and business processes, but this provides no guarantee to the client of proper design, rules, or implementation with regard to asset protection in a data center. The recent unfortunate incident, which affected infrastructure with SECNUMCloud (an initiative by the French National Cybersecurity Agency), ISO-27001 and even HDS certification, demonstrates this. Certification of compliance with an ISO, HDS or SECNUMCloud standard is by no means a guarantee of the physical security of a data center.
Regarding asset protection, in France, APSAD certification delivered by insurance companies and the CNPP (National Center for Prevention and Protection) provides a guarantee of reliability and effectiveness for asset security. This certification is based on reference standards and requirements, stemming from experience of incidents, which apply to the design of the facility, desired results, training of staff, and maintenance. This voluntary certification is extremely strict, costly, and demanding. For us, it represents the minimum requirement for a data center, bearing in mind the sensitivity of the assets hosted. This APSAD certification is a guarantee of security.
The risk is very low if it is correctly addressed by measures built into the data center.
In our experience, inverters and batteries represent the highest risk.
Over the past ten years, across its data centers in France using around 40MW, Scaleway has experienced one battery fire, on September 16, 2019.
and four inverter explosions
On June 27, 2013, the neighbor of our DC2 data center, specialized in recycling paper, burned down:
More recently, on June 2, 2019, the neighbor of our DC5 data center, specialized in chemical processing, also burned down just a few meters away from our premises:
To be very clear, running a data center means anticipating and managing risk. This risk may be internal or external, and it is real.
Our job is to plan for every eventuality, no matter how unlikely. Each time, the design of our data center and our automatic mechanisms worked perfectly and prevented your assets from being affected by a major incident, with no outages.
Scaleway’s approach is based around three objectives:
This approach is applied by all data centers around the world. In this sector, not only asset protection but also business continuity is paramount, even in the unlikely event of a fire.
Passive protection (built-in)
We divide each data center into compartments, all of which are fire-resistant. The walls, flooring, ceilings, doors, and windows are designed to resist fire and prevent it spreading to the rest of the building. The surface area and duration of this resistance depends on the risk involved.
In other words, if there is an incident within one compartment, it will not spread to the rest of the building for at least one or two hours.
For example:
Ventilation ducts are shut off by valves which close automatically in case of fire to stop it spreading. Cable passages are caulked and treated with mastic and intumescent paint.
Our level of fireproofing is ensured by mineral wool sandwich panels with fire properties corresponding to APSAD standard D14-A, or concrete with a certain thickness, and specially designed doors.
Smoke is just as dangerous as fire itself. Each compartment has a smoke extraction system which is able to work in 400ºC heat for two hours.
With regard to the neighbors of our data centers (which have caused problems on two occasions throughout our history), where we are not able to implement a 10-meter distance, we protect our centers with fire protection walls, heavy-duty roads and fire hydrants which allow the emergency services to safely intervene.
Finally, since DC3, we install all high-risk equipment such as power generators and high-voltage transformers outside the building.
Active protection
Our data centers are all standard equipped with fire detection systems corresponding to APSAD DC7 or N7.
These are highly advanced systems. They work by taking air samples and are unaffected by the significant air currents present in data centers. These are reliable early detection systems, which can detect a fire in under 40 seconds. The manufacturer carries out maintenance twice per year, which also has specific certification (APSAD D7).
The first intervention or check in the event of a potential detection is carried out by a fire safety agent specially trained in firefighting (SSIAP2), present 24/7 at all data centers, and our technicians. They use installed fire extinguishers which are certified APSAD N4 or fire hose cabinets installed in the storage spaces of DC2, DC3 and DC5.
N.B.: Our APSAD DC7, N7 and N4 certification and periodic maintenance certification (APSAD Q7 and Q4) are available by request from technical support.
Fire suppression
Our sector requires service and operating continuity even in the event of a fire. There are two main types of automatic fire suppression systems on the market which can put out a fire without interrupting services:
As for sprinkler systems, which are a specific requirement in the USA, these do not allow operating continuity.
We use both of the above systems:
We avoid using gas systems due to a number of hard drive incidents which have occurred over the past few years, caused by the noise made by these systems(1)
In light of the unjustified disinclination of the market toward water mist systems, we tested them in real conditions in June 2012, in conjunction with the CNRS (French National Center for Scientific Research) and in the presence of our clients, to measure their effectiveness in extinguishing a fire without damaging IT equipment:
We also tested this approach on live transformers in September 2012:
Since then, this automatic water mist fire suppression system has become widely used in almost all data centers around the world, and is recognized as the most effective system.
N.B.: Our FM, VdS/OH1, DIFT and APSAD R13 certification and periodic maintenance (APSAD Q13) certification is available by request from technical support.
Facilitating emergency services intervention
Our buildings and rooms are built with fire-resistance in mind, to allow safe intervention by the emergency services if an incident occurs. They are made of concrete or fire-resistant mineral wool sandwich panels.
To raise the alarm, DC3 is equipped with a specific priority telephone (TASAL – Automatically Monitored Line Telephone) installed by the fire department.
All our data centers have a limited height (maximum 11 meters), are equipped with fire hydrants, heavy-duty roads and a fire water run-off collection system in line with regulations.
Audit
We are proud of our data centers and their security. We consider that we have implemented the best solutions to protect your most valuable asset: your data. We are well aware of the huge responsibility this represents. There can be no compromises when it comes to your data.
For this reason, and because of the high level of protection implemented in our data centers, we are covered by the best insurance on the market.
Our four data centers are audited by our insurer at least once per year. They can also be audited by you, our client, accompanied by experts chosen by you.
Our certification, risk analyses, and safety information can be consulted and audited upon request. We only charge for the time our teams spend assisting you and putting together the required technical files.
We regularly organize visits to our data centers, particularly on heritage days, and we would be delighted to welcome you into the heart of our infrastructure as soon as the health situation allows.
Scaleway’s approach in other data centers
All our data centers in France belong to Scaleway, and we also have colocation data centers in the Netherlands and Poland.
These data centers have not been designed by, and are not run by Scaleway, rather we work with Iron Mountain and Equinix.
We have a long-term contract with these partners, and we regularly audit their sites to ensure they apply similar criteria as for our own sites in terms of infrastructure availability and asset security.
The APSAD certification and reference standard does not exist outside of France, but each country has similar technical reference standards that many data centers follow and adhere to, such as VdS.
N.B.: The certification for our colocation partners is available by request from technical support.
One night in September, a power transformer shut down in one of our Parisian data centers. Read on to find out what happened during this tense night.
Discover how DC5 operates: How do we optimize the energy footprint of datacenter? How can we prevent power loss? How can we deliver proper cooling make it modular?
When you build your infrastructure with Scaleway, it’s important to take a few simple rules into account, to limit the risk of data loss. Data is a shared responsibility: provider & customer.