AWS goes down due to a power outage and is notified that 'some hardware may not be recoverable'

by Web Summit

Amazon Web Services (AWS), which provides a wide variety of infrastructure services around the world, holds 32% of the global market share in cloud services. On December 22, 2021, AWS was reported to be down due to a power outage, affecting services such as Slack and Asana.

AWS power failure killed some hardware and instances • The Register
https://www.theregister.com/2021/12/22/aws_outage/

AWS is an indispensable infrastructure for enterprises, but a large-scale failure occurred in December 2021. It has influenced numerous services such as Nintendo, Netflix, Disney +, Coinbase, a cryptocurrency exchange, and Slack, a communication tool.

Official Explains AWS's Massive Failures Affecting Nintendo and Netflix-GIGAZINE

Apart from the above, in November 2021, a vulnerability ' CVE-2021-44228 (Log4Shell) ' that was present in Java's Log4j library and enables remote code execution was discovered, and this is an AWS account hijacking. AWS system administrators are faced with a number of difficulties, such as being pointed out that they are available.

How to hijack AWS account with 'Log4Shell' is reported --GIGAZINE

Then, on December 22, 2021, it was reported that a power outage occurred in the USE1-AZ4 Availability Zone in the US-EAST-1 region in northern Virginia, USA, and the system went down.

The power outage occurred at 4:35 am Pacific time. AWS has detected that Amazon Elastic Compute Cloud 's IaaS service is experiencing launch failures or network issues. Twenty-six minutes later, following a power outage, AWS recommended customers to move their workloads to another cloud that was still powered.

At 5:39, power was restored and service gradually began to be restored, but at 6:51, it was reported that 'network problems were preventing a complete recovery.' Services such as Slack, Asana, and Trello were affected by this failure.

Trello is down and down. As shown in the graph. The source is https://t.co/uvycqdcJ8n pic.twitter.com/YY5jey8Y04

— Senri High School News Department / Alumni Association --Kansai Committee 63 (@ins_magazine) December 22, 2021

In addition, the AWS status report states, 'As is often the case when power goes down, some hardware may not be recoverable, or the affected EC2 instance or EBS volume may be completely recovered. It may not be possible. At present, I can't say for sure, but I think it's difficult to recover all the small number of EC2 instances and EBS volumes that are being recovered. '

As is often the case with a loss of power, there may be some hardware that is not recoverable, which will prevent us from fully recovering the affected EC2 instances and EBS volumes. We are not quite at that point yet in terms of recovery, but it is unlikely that we will recover all of the small number of remaining EC2 instances and EBS volumes.

AWS also reported a failure in the US-EAST-1 region in September 2021 and in the US-WEST-1 region on December 15. AWS encourages its customers not to rely on a single Availability Zone.