Alibaba Cloud data center carefully dried after being soaked by firefighting efforts

by Charles Chan

A fire broke out at an Alibaba Cloud data center due to an explosion of a lithium-ion battery, causing a network outage. The firefighting efforts caused the equipment in the data center to become flooded, so Alibaba Cloud is currently removing the wet equipment from the data center, drying it out, and rushing to restore operations.

status.alibabacloud.com/#/eventDetail?eventId=21
https://status.alibabacloud.com/#/eventDetail?eventId=21

Alibaba Cloud waiting for hardware to dry out after fire • The Register
https://www.theregister.com/2024/09/17/alibaba_cloud_singapore_fire_recovery/

Alibaba cloud servers being 'carefully dried' after firefighter drenching last week | Tom's Hardware
https://www.tomshardware.com/desktops/servers/alibaba-cloud-servers-being-carefully-dried-after-firefighter-drenching-last-week

At 11:20 a.m. on September 10, 2024 (Japan time), Alibaba Cloud's monitoring system detected an anomaly in network access in Zone C of the Singapore region. After an investigation by Alibaba Cloud engineers, it was determined that a fire had broken out in a Singapore data center due to an explosion of a lithium-ion battery. By 10:55 a.m. on the same day, the fire had already been put out by firefighters.

After that, at 12:30, Alibaba Cloud reported that disaster recovery for cloud products such as Cloud Database Redis / MongoDB / RDS MySQL, OSS, OTS, etc. was successfully completed. However, at 14:40, Alibaba Cloud reported that 'the fire situation in the affected data center is basically under control, but the temperature inside the facility is still high.'

As of 8:04 p.m., the fire alarm had not been completely lifted, and it was discovered that some of the network equipment in the data center was experiencing abnormalities due to high temperatures. Then, at 1:46 a.m. on September 11, Alibaba Cloud reported that it had performed an emergency power cut in the server building in Zone C, stating, 'At the request of the local fire department, water spraying for firefighting activities is continuing, causing water to accumulate in the server room, resulting in leakage and creating a risk of electrical shorts.'

'The fire was linked to lithium-ion batteries stored in a battery room on the third floor of the four-storey building. We used four water jets to extinguish the blaze and also deployed an unmanned firefighting robot,' the Singapore Civil Defence Force (SCDF) said in a statement to local media.

Firefighting efforts continued for a full day, and Alibaba Cloud engineers were finally allowed to access the first floor area of the data center at 17:25 on September 12. After that, the safe migration of hardware equipment was carried out, but as of 15:00 on September 16, some areas of the data center were sealed off and could not be entered.

Alibaba Cloud said, 'Some of the affected hardware and machines are located in dangerous, blocked off areas of the building where access is prohibited, and some hardware and machines need to be carefully dried to ensure data security.' The migrated equipment has also been carefully dried, wired, powered on, and verified.

When checking the status of the Singapore data center, we found that some services were still experiencing network issues at the time of writing.

Alibaba Cloud Health Status
https://status.alibabacloud.com/#/?region=ap-southeast-1

The fire also caused disruptions to some of the services that rely on Alibaba Cloud, including Alibaba's e-commerce platform Lazada and TikTok.