Northflank Tells Customers to Rotate All Secrets After Breach

Developers who host apps on Northflank have been told to assume their stored secrets are gone. The company says a security breach led to unauthorized use of customers' AI API keys.

What Northflank told customers

Northflank is a cloud platform for deploying and managing applications, containers, databases and AI applications. It disclosed the incident on October 30 and asked customers to rotate their environment variables and API keys at once. The alert came after cloud security company Wiz sent a threat warning.

Soon afterwards the advice got sharper. In an October 30 update the company wrote: "Assume all environment variables have been decrypted. Disable/rotate all keys."

Later Northflank confirmed that several customers had seen their AI keys used by someone else. Customers should also check usage and billing logs at their AI providers for unexpected activity and report anything suspicious to both Northflank and the provider.

Why environment variables matter

Applications often keep database passwords, cloud credentials, authentication tokens and API keys in environment variables. If those leak, attackers can reach other services the customer uses. A stolen API key can also mean unauthorized charges.

Timeline and fixes

According to the incident timeline Northflank published, Wiz raised the alarm at 11:43 UTC on October 30. Engineers looked at it a minute later. A network policy patch followed, then more patches at 12:50 UTC and 13:05 UTC. The company also asked Wiz's incident response team for help.

Northflank turned on credential rotation for PostgreSQL, MongoDB, MySQL and Redis. It switched on audit logs for every customer, free accounts included. It still recommends rotating secrets kept in workloads, secret groups and global secrets, and closing public access to add-ons where it is enabled.

A wider pattern of attacks

In a report called "Black October for AI Developer Platforms", Wiz counted at least nine security incidents at AI development platforms during the month. Northflank, Railway, OpenCode, Warp and Lightning AI are among them. Railway's incident also involved possibly exposed decrypted environment variables. OpenCode disclosed exposed AI API keys. Warp reported a breach of AI agent logs and user telemetry data.

Wiz says attackers in some of these cases went after environment variables holding AI service credentials. The tactic is called LLMJacking: using stolen credentials to call AI models on another organization's account. That can run up large bills or let the access be resold. In two cases Wiz investigated, compromised enterprise credentials produced about 2.5 million Amazon Bedrock API calls and roughly $100,000 in usage.

Wiz has not confirmed that one group is behind all nine incidents. Northflank said the customer reports may be tied to the attacks Wiz described.

Several things are still unknown. Northflank has not said how attackers entered its systems, how many customers were hit, or whether anything besides AI keys was abused. It did not answer questions from the outlet that reported the story.

Our take

The Northflank case suggests that AI keys are becoming a favorite target because they convert directly into money. Teams that keep such keys in hosted platforms may want to limit what each key can do and to watch spending alerts. It would help to see Northflank's full account of the intrusion once it is ready.

Source: BleepingComputer