AWS root credentials compromised - €30k bill

A new client reached out in a state of urgency after discovering several unusually large EC2 instances running in their AWS environment. The instances were consuming significant resources, and the client was unsure who had launched them or why they existed.

Upon gaining access, an immediate investigation was launched. The audit revealed that the instances had been created using the root account - a red flag in itself, as use of the root account in daily operations is strongly discouraged. The instances were GPU-optimised and typically suited for high-performance computing workloads, such as video rendering or machine learning. In this case, the unauthorized use pointed to a more sinister culprit: cryptocurrency mining.

Cryptocurrency mining workloads are known for leveraging GPU-intensive instances due to their ability to perform parallel computations efficiently. The instance type of the active instances aligned with this type of activity.

The first step was to secure the account. Root credentials were rotated immediately, and a full AWS CloudTrail review was performed to assess the scope of the incident. Particular attention was paid to whether the attacker had created any additional IAM users, issued new access keys, or escalated privileges in ways that might enable persistence beyond the original intrusion. Network access patterns and IAM policies were reviewed in detail to detect any lateral movement or hidden footholds.

An important aspect of AWS networking came into play during remediation. While security groups are commonly used to restrict inbound and outbound traffic, they do not interrupt established connections. To fully isolate the compromised instances and terminate any ongoing activity, network ACLs (NACLs) were used to block traffic at the subnet level - ensuring all active sessions were forcibly dropped.

After isolating the systems, a report was generated from CloudTrail to enumerate all actions performed by the attacker. The evidence confirmed that the primary activity was spinning up high-cost EC2 instances for mining purposes, with no additional infrastructure or credential modifications detected. The instances were terminated, and snapshots were taken of both EBS volumes and memory for future forensic analysis if desired.

With the immediate threat neutralised, attention turned to cost recovery. A support ticket was submitted to AWS, including details of the compromise, the remediation steps taken, and confirmation that the security posture had been improved. AWS reviewed the case and agreed to waive the charges associated with the unauthorised usage.

The root cause of the breach was traced back to a common but serious misstep: a developer had embedded AWS root access keys in application code that was later pushed to a public code repository. Publicly exposed credentials are actively scanned by automated tools, and such leaks are among the fastest paths to account compromise.

While the incident had the potential for severe financial and operational impact, prompt investigation, comprehensive remediation, and a cooperative engagement with AWS support helped convert a major breach into a valuable learning opportunity. The client implemented stricter credential management practices and now follows least-privilege principles across their accounts.