The detailed timeline of the criminal's attack method and countermeasures is released as to why personal information was leaked from Stack Overflow
Regarding the personal information leak of the technical community
A deeper dive into our May 2019 security incident --Stack Overflow Blog
https://stackoverflow.blog/2021/01/25/a-deeper-dive-into-our-may-2019-security-incident/
Around midnight, May 12, 2019, several members of the Stack Overflow community warned that their new account was gaining unexpected privileges. The Stack Overflow team discovered that a completely unknown user had moderator and developer-level access to all sites within the Stack Exchange Network and immediately suspended their accounts. After the damage caused by the leakage of the source code and account information of 184 people, the team started the process of identifying the cause and auditing.
The team investigates a database of external traffic to capture the attacker's footsteps. Based on the attacker's account identifier, IP address, information from the customer support team, etc., the 'footprint until the attacker gains privileges' reproduced by the team is as follows.
・ Thursday, April 30, 2019
Attacker launches Stack Overflow infrastructure investigation. In particular, I focused on web servers that host build source code control systems and development environments.
・ May 1, 2019 (Wednesday)
An attacker attempted to access a Stack Exchange chat room internally used by the Stack Overflow SRE team, but failed.
I received an email from a person claiming to be a 'Stack Overflow customer' asking me to share the Stack Overflow source code for auditing purposes. As a general rule, Stack Overflow did not publish the source code, and Stack Overflow rejected the request because it could not confirm that the sender's email address belonged to the customer.
The attacker then creates a team on Stack Overflow and invites another account. At this time, each account was authenticated on a different device.
・ May 2, 2019 (Thursday)
The attackers have viewed a number of 'Posts related to
In addition, there was another support request following the request the day before, but this time it was a spoofing of the actual customer's email address. Also, the content was based on a case study viewed on Stack Overflow. An automatic anti-spoofing reply was sent to the real customer, immediately informing them that it was not a legitimate request.
・ May 3, 2019 (Friday)
Stack Overflow rejected the May 2 support request. The attackers continued to investigate the infrastructure.
・ Saturday, May 4, 2019
An attacker attempted to download the Stack Overflow source code on GitHub Enterprise, but was redirected to the login screen. However, at this time, a later investigation revealed that the URL of the private repository was mistakenly included in the public repository.
・ May 5, 2019 (Sun)
This day was a day when attackers were active. The attacker successfully bypassed access restrictions and logged in to Stack Overflow's development environment. At first, it seems that it was not possible to do so much exploration due to lack of privileges, but the 'account spoofing function' that Stack Overflow had prepared for testing was discovered, and the attacker included privileges as a website moderator. You have gained 'Community Admin' privileges.
The attacker also tried to gain 'developer' privileges by using the account recovery function, but failed because he could not receive the email required for recovery. However, there was a route in the development environment that even community administrators could receive emails, and the attacker used that route to receive emails and finally succeeded in stealing 'developer' privileges.
・ May 6, 2019 (Monday)
The attackers continued to investigate the development environment. I used the acquired developer privileges to access the production environment and engaged in activities such as collecting information and trying out spoofing functions. This attempt was unsuccessful because the spoofing feature was not implemented in the production environment.
As the attacker explores the development environment, he discovers access to TeamCity . Access rights that were no longer in use remained. The attacker succeeded in gaining access to TeamCity and immediately gained administrator privileges on the build server due to a misconfigured role assignment.
The attacker didn't seem to know much about TeamCity and was browsing questions about TeamCity on Stack Overflow. The attacker has been browsing Stack Overflow questions many times since then, explaining that this browsing history helped the Stack Overflow team understand how the attacker was attacking.
At the end of the day, the attacker attempted to access the data center's build agent, but failed because a VPN connection was required.
・ May 7, 2019 (Tuesday)
The attackers investigated how to set up various internal sites, including how to set up GitHub Enterprise used on Stack Overflow.
・ May 8, 2019 (Wednesday)
The attacker logs in to TeamCity and explores the build server's file system. I got an SSH key to access my internal GitHub Enterprise. The attacker also used this SSH key to download important Stack Overflow source code while also obtaining Wiki information about scripts and operations for development setup. I also tried to access the GitHub Enterprise website, but it failed because it had two-step authentication set up.
The attacker was also reading a question about how to build a .NET project on Stack Overflow.
・ May 9, 2019 (Thursday)
The attacker first downloaded the latest source code. This action will be the daily routine of future attackers. The attacker attempted a VPN connection from a
・ May 10, 2019 (Friday)
The attackers continued to investigate web servers and applications on Stack Overflow.
・ May 11, 2019 (Sat)
The attacker impersonates a Git user, creates a new project in TeamCity, and tries to build it several times. After that, I tried to copy the database containing the settings of Stack Overflow's local environment, but it was designed so that it could not be copied externally, so the attacker created Gist on GitHub inside Stack Overflow and copied the database from there. .. You have successfully modified the production database that stores your Stack Exchange Network data.
After completing a series of work, the attacker erases all footprints. However, TeamCity's build information remained in the 'Recycle Bin', and it is said that the Stack Overflow team was able to grasp the situation later.
・ May 12, 2019 (Sun)
On May 12, the day Stack Overflow was attacked, the community reported a problem after the attacker executed the SQL, and Stack Overflow's incident response team began investigating.
At this point, the scope of the attack was unknown, so the team first deleted the attacker's Stack Exchange Network account. After that, it turned out that TeamCity was also hacked, so I immediately took TeamCity offline.
After that, the team was affected by the attack by deleting the 'route that even community administrators can receive emails' and 'vulnerable website settings used to hack Team City' used by the attackers. I deleted or reset my account. We also asked the secondary response team for forensics .
The attacker tried to spoof the account and access TeamCity during this time, but failed due to the above measures. However, the Stack Overflow team was unaware of the source code leak at this point, and the attackers were still able to download the source code.
・ May 13, 2019 (Monday)
Attackers continued to download the source code and attempt to access Teams City. The Stack Overflow team parses the traffic logs and changes all access keys and leaked credentials in the development environment. In addition, vulnerability countermeasures have been taken, such as closing the holes that existed in the access authentication to the development environment and making the authentication information unreadable.
The attackers continued to try to access Teams City and continue to download the source code. The attacker was also browsing Stack Overflow questions about how to programmatically update a number of Git repositories and how to create a SQL database.
・ May 14, 2019 (Tuesday)
The Stack Overflow team finally confirmed that the SSH key on GitHub was leaked on this day. I immediately moved the GitHub server behind the firewall to block outside access. When I audited the GitHub repository, fortunately it wasn't tampered with.
The Stack Overflow team is paying attention to the fact that once you add an SSH key to GitHub, you can use that key to log in without the need for two-step verification. You can see how important SSH key management is.
Eventually, Stack Overflow contracted with an external security vendor to request an audit and double check.
・ Thursday, May 16, 2019
We have established a policy of continuing to audit attacks and, if possible, notify users affected by the attack.
Attackers were less active on this day and only browsed questions about SQL databases in Azure on Stack Overflow.
・ May 17, 2019 (Friday)
The attacker was browsing a question about SQL and authentication on Stack Overflow.
The Stack Overflow team announced today that 250 users are suspected of leaking personal information.
・ Saturday, May 18, 2019
The attacker is investigating 'How to delete a remote Git repository' on Stack Overflow, but nothing else is active.
・ May 22, 2019 (Wednesday)
The Stack Overflow team has notified users of leaked personal information.
・ May 23, 2019 (Thursday)
The Stack Overflow team has completed a secondary investigation and has created a set of remedies to address the underlying issue that caused the attack.
In response to this attack, the Stack Overflow team moved the build and source code control system inside the firewall, and took measures such as reviewing private key management. It also emphasizes the importance of increasing security by recording all internal traffic and using two-step authentication to prepare for attacks.
Related Posts:
in Web Service, Security, Posted by darkhorse_log