Empty S3 buckets cause AWS bills to explode, Amazon announces it will address the issue
How an empty S3 bucket can make your AWS bill explode | by Maciej Pocwierz | Apr, 2024 | Medium
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
As a proof of concept for a document indexing system he was building for a client, Pocwierz created a single S3 bucket in the AWS ' eu-west-1 ' region and uploaded several files for testing. After that, to make sure that what he was doing on AWS was within the free tier limits, he checked his AWS bill, which turned out to be over $1,300. When he checked, he discovered that about 100 million S3 PUT requests had been made in just one day.
By default, AWS doesn't log requests made to an S3 bucket. However, you can change the settings to log them using
Pocwierz, who wondered why some third parties were sending a large number of fraudulent requests to the S3 bucket, conducted an investigation. It turned out that this was not a DDoS attack, but rather an excessive number of requests caused by a 'popular open source tool that has a default configuration to store backups in S3.' In other words, every deployment of a tool using this open source tool tries to store backups in an S3 bucket. Pocwierz said, 'It seems that a large number of misconfigured systems tried to store data in my private S3 bucket, which resulted in excessive billing.'
When Pocwierz contacted AWS support, he received the following response: 'Yes, S3 charges for invalid requests. This is expected behavior.'
Therefore, if someone opens a terminal and types the following, the error message 'AccessDenied' will be displayed, but the user will still be charged for the request. Note that this is a request that can be made without an AWS account.
[code]aws s3 cp ./file.txt s3://bucket name/random_key[/code]
In addition, Pocwierz is also curious about the fact that, even though he created an S3 bucket in the 'eu-west-1' region, half of the requests are coming from the 'us-east-1' region. 'S3 requests that do not specify a region will default to us-east-1 and will be redirected as necessary. This will force bucket owners to pay additional fees for redirected requests,' he said.
The reason for the high AWS bill is as mentioned above, but Pocwierz wrote, 'I had another idea I wanted to consider.' The idea he wanted to consider was, 'If all the misconfigured systems were trying to back up their data to an S3 bucket, what would happen if we just left them there?' So Pocwierz set up the bucket for public writing, and within 30 seconds, he was able to collect over 10GB of data. 'Of course, I can't reveal who wrote what data, but I was surprised that an innocent oversight of the configuration could lead to a dangerous data leak,' Pocwierz said.
As mentioned above, Pocwierz is not naming the open source tool in question, as disclosing its name could expose affected companies to the risk of data leakage.
Based on these findings, Pocwierz said he learned the following three lessons from using Amazon S3:
Anyone who knows the name of your S3 bucket can increase your AWS bill however they like.
There is no way to prevent this other than deleting the bucket. If it is accessed directly through the S3 API, services like CloudFront or WAF cannot protect the bucket. A standard S3 PUT request costs just $0.005 per 1000 requests, but a single machine can easily perform thousands of requests per second.
◆2: Adding a random suffix to bucket names can improve security
Adding a suffix reduces the system's vulnerability to misconfigurations and deliberate attacks. At a minimum, avoid using short, generic names for your S3 buckets.
◆3: When making large requests to S3, you should always explicitly specify the AWS region
By specifying a region, you can avoid additional costs due to S3 API redirects.
Pocwierz contacted the administrator of the open source tool in question with his findings. This resulted in the default configuration being quickly fixed, but it does not fix existing deployed use cases. Pocwierz also notified the AWS security team of the problem, suggesting that they 'restrict S3 bucket names that may be problematic to protect users from fraudulent charges and data leaks.' However, AWS was reluctant to address misconfigurations in third-party products. AWS canceled the huge charges incurred by this issue, but emphasized that this was 'exceptional.'
In response to reports of the problem, AWS' Jeff Barr said, 'We appreciate everyone who brought this article to our attention. We agree that customers should not have to pay for fraudulent requests that they did not initiate. We will provide more details soon about how we are working to prevent these charges.' He emphasized that the company is taking measures to address this issue.
Thank you to everyone who brought this article to our attention. We agree that customers should not have to pay for unauthorized requests that they did not initiate. We'll have more to share on exactly how we'll help prevent these charges shortly. #AWS #S3
— Jeff Barr ☁ (@jeffbarr) April 30, 2024
How to empty S3…
◆ Forum is currently open
A forum related to this article has been set up on the official GIGAZINE Discord server . Anyone can post freely, so please feel free to comment! If you do not have a Discord account, please refer to the account creation procedure explanation article to create an account!
• Discord | 'Have you ever incurred unintended usage fees on AWS, GCP, etc?' | GIGAZINE
https://discord.com/channels/1037961069903216680/1235525804633489448
Related Posts:
in Web Service, Posted by logu_ii