Railway postmortem traces two routes to stolen AI keys
By AgentRiot Editorial
Railway details a build-system escape and a cloud identity error, narrows what its exposure estimate means, and pledges to cover unauthorized AI usage.

A build credential that could do too much, followed by a cloud identity configuration that trusted the wrong requests, opened separate routes into Railway during its October security incident. In its November 10 postmortem (opens in new tab), the hosting company describes how the attacker reached build histories, obtained Railway's source code and later assembled the permissions needed to decrypt customer secrets.
Railway estimates unauthorized usage of stolen keys reached about $740,000. It also reports exposure of linked-account tokens for 1,852 users. Its larger figure, about 100,000 workspaces, remains an upper bound on possible environment-variable decryption, not a count of workspaces with confirmed stolen secrets.
The report, last updated November 10 at 18:02 UTC, supplies mechanisms missing from the preliminary account AgentRiot covered in October (opens in new tab). Those findings remain Railway's account. A specialist forensic firm's investigation is ongoing, and the company has not yet released public indicators of compromise.
The first route ran through the builder
Railway says the attacker began creating accounts on October 18, but those early attempts failed. On October 20, probing reached the build infrastructure. All times below are UTC.
At 07:28, a previously unknown flaw let the attacker partially escape filesystem isolation and read some files on a shared build host. This was not described as unrestricted control of every neighboring build. The consequential discovery was an internal credential used by build servers, including to push completed images.
That credential was supposed to authorize build work. Railway says its self-hosted Temporal system also accepted it as a trusted client. Temporal coordinated asynchronous build tasks whose stored histories contained repository-cloning tokens, environment variables and build metadata. The attacker could therefore use a credential taken from the builder to read histories belonging to other builds. According to the first-entry-point account (opens in new tab), the failure crossed both an isolation boundary and an authorization boundary.
One accessible history belonged to Railway's own support service, built from its internal monorepo. Access to build-cache repository snapshots then gave the attacker a copy of that monorepo. Railway says it has no indication that customer repository snapshots or repositories were downloaded. Its own source code, however, exposed internal architecture and helped the attacker search for additional routes.
Anomalous registry calls paged the on-call engineer at 13:58. Railway halted builds at 17:03 and revoked and replaced the build credential at 17:28, while restricting which services builders could reach. It patched the isolation flaw and later resumed builds. The report says subsequent attempts to escape the patched builder failed. That did not remove the source code already obtained.
A second route combined cloud access and decryption authority
At 03:26 on October 21, Railway says an erroneous OpenID Connect binding let the attacker obtain a short-lived cloud credential through infrastructure associated with its deployment pipeline. The company does not publish the precise binding error. The second route should not be collapsed into another successful builder escape.
The short-lived credential allowed access to internal provisioning and deployment credentials. Those, in combination, let the attacker reach an orchestration machine, enter a private network and read encrypted environment variables from a production database. The machine also held longer-lived authority needed to decrypt the per-environment keys protecting those variables.
The attacker retrieved encrypted envelopes between 04:07 and 05:21. Railway's secondary-entry-point timeline (opens in new tab) records decryption activity during two intervals: 04:42 to 05:14, and 06:36 to 07:15. Its customer-impact section explicitly corrects the earlier account: decryption began at 04:42, not after 05:27.
Railway records credential disabling at 07:14, although the detailed activity interval ends at 07:15. It disabled the decryption service and halted builds again at 07:22. The report's opening puts full eviction at 09:24; its detailed narrative says no customer-data access continued beyond 09:25. Railway says customer-data access had ended by 09:25, a conclusion it validated later that day.
Envelope encryption did not require the attacker to steal the root key. Railway says that key stayed inside its upstream cloud provider's key-management service. The attacker instead obtained authority to have encrypted environment keys decrypted. Google Cloud's explanation of envelope encryption (opens in new tab) describes the same distinction between encrypted data keys and the key that wraps them. Railway does not identify its upstream provider here.
Three exposure counts answer different questions
Railway's customer-impact findings (opens in new tab) separate several affected surfaces:
- At most 1.85 percent of workspaces, about 100,000, could have had environment variables decrypted. Available logs identify environments, not individual variables, making this an upper bound.
- Stored histories of 353 builds were read between 13:50 and 15:30 on October 20. These included sealed variables.
- Decrypted linked-account data included bring-your-own-key agent credentials and GitHub OAuth tokens for 1,852 users. Railway says it reset those tokens, notified each user and identified no Railway account logins using credentials taken in the incident.
These are different units and routes of exposure. They should not be added together as a total victim count. Nor does the lack of identified Railway account logins mean stolen provider keys were unused elsewhere.
Railway says database queries prioritized AI-provider credentials and customers with large numbers of variables. An additional query matched cloud, database and payment credentials. It reports no forensic indication that customer databases were accessed. That is narrower than saying no customer data could have been reached through any stolen credential.
Revocation notices also covered more than the forensic upper bound. Starting at 09:25 on October 21, Railway asked providers to revoke potentially compromised keys. Because it could not yet isolate exposure, it supplied hashes of every key it held; some providers revoked additional keys at their discretion. A provider warning therefore does not prove that a particular key was decrypted.
AI-key resale is evidence of misuse, not an attacker identity
Railway's attack-intent assessment (opens in new tab) rests partly on queries specifically seeking AI credentials. It believes the keys were passed to relay or resale services for distillation. Customer-supplied request examples showed coding-agent traffic, reviews of agent behavior for training data, and checks of training tasks and scoring systems.
Those observations support a market for stolen model access. They do not establish who operated the resale services or whether the original intruder supplied keys to other parties. Railway explicitly leaves that relationship unresolved.
In its separate Black October investigation (opens in new tab), Wiz describes stolen enterprise AI credentials being tested and subsequently used at scale, with links to resale operations. Wiz also says it cannot definitively establish the relationship between those operations, the initial compromises and the bulk abuse. Its published indicators are not independently established Railway intrusion indicators.
The tooling claim has a similar limit. Railway describes models with safeguards removed and a harness similar to ARTEX, citing Wiz. That is a claim about tools and attack speed, not identification of a named operator, country or common actor behind every October breach. The report does not quantify how much faster the attack was because of AI.
Running containers did not mean every application stayed healthy
Railway kept existing deployments running during the build pauses. But its recovery account (opens in new tab) acknowledges that some customers had downtime: revoked credentials broke applications, and paused builds prevented them from deploying replacements. A process remaining online was not enough to keep its provider integrations working.
Storage-bucket rotations also left some running containers with old credentials, stopping database backup archiving until Railway redeployed them. Sessions and API access were disrupted during recovery. These were operational effects of containment, not merely an inconvenience when shipping new features.
Builds returned in stages on October 22: Enterprise at 03:32, Pro at 06:29, Hobby at 21:08, and Free and Trial at 22:49. The public status history (opens in new tab) records those restorations. Railway says accelerated migration to separate build infrastructure and a surge in customer rotations strained capacity, followed by slow or failed builds. Its November report says signups reopened November 2 and the static-egress exception to service SSH restoration ended November 4.
On delayed customer notification, Railway says it prioritized provider revocation while determining impact and removing access, partly to avoid newly rotated secrets being stolen again. It now promises a definite period for assessing impact, after which broad notification will occur regardless. The report does not specify that period's length.
What changed in the architecture
Railway says each build now receives a dedicated ephemeral virtual machine (opens in new tab), retaining the previous isolation layer underneath. Its credential can push only one specific container image. The shared systems exploited in the first route are no longer used.
This was an accelerated rollout of infrastructure already being tested, not a new builder designed during the attack. Railway's internal hardening team found another exploit in the old builder within an hour of the first patch (opens in new tab). That helped drive migration of all users to the separate compute and storage system. Failed attacker retries against the first fix were not enough to establish that the old architecture was safe.
It also says it rebuilt CI with finer-grained access controls, restricted network paths and stronger approvals in critical areas. Decryption is now concentrated in a secrets service: other services cannot obtain keys that decrypt data-encryption keys, callers are limited to credentials needed for their work, and encryption and decryption operations are centrally logged.
The architectural lesson is specific. A builder should not acquire authority over other builds merely because both use the same internal identity system. A database ciphertext export should not be enough when combined with permissions already available on a broadly privileged orchestration host. Railway's changes aim to interrupt those combinations, rather than relying on encryption at rest alone.
Cloud federation needs equally narrow trust. Google's deployment-pipeline guidance (opens in new tab) warns that shared token issuers require conditions restricting accepted organizational identities. That is a useful review principle for operators, not a reconstruction of Railway's undisclosed configuration. Short credential lifetimes cannot compensate for granting a credential to the wrong requester.
What customers can still do
Customers who have not completed remediation should revoke old secrets at their issuers, replace the values stored in Railway and redeploy dependent services. Railway's status guidance (opens in new tab) says customers outside its higher-risk notification groups are not confirmed safe; it also says completed post-containment rotations do not need repeating merely because of that update.
Its bucket and GitHub recovery guide (opens in new tab) distinguishes reconnecting GitHub from revoking registry tokens at GitHub itself. Removing a token from Railway did not invalidate it at its issuer. Redeploy services using rotated bucket credentials and check that integrations and backups actually work. For missing projects after reauthentication, the login recovery guide (opens in new tab) directs customers to use their original account identity and contact support rather than populate an empty replacement account.
For unauthorized charges, Railway's November 10 refund update (opens in new tab) says AWS Bedrock and Google Gemini account teams had initiated fraud refunds. It describes OpenAI and SpaceXSI reviews as case-by-case. It says Railway would cover Anthropic breached-key costs while discussions continued and breach-related OpenRouter costs after OpenRouter declined refunds. These are Railway's dated descriptions and commitments, not blanket provider refund guarantees. The roughly $740,000 usage estimate is not a final loss or a completed-refund total.
Older guidance has not fully caught up with that promise. The preliminary report's October 26 update (opens in new tab) said Railway was unable to refund third-party spend. Its claim form (opens in new tab) still says Railway is not responsible for usage of the keys. That disclaimer does not establish that the later backstop was withdrawn, but customers still need clarity on eligibility and payment timing.
Railway links the form for provider, timing, estimated charges and redacted evidence, with support at Railway Station (opens in new tab) for follow-up. OpenAI customers are directed to open a provider case and pass its ID to Railway. The form does not revoke keys or submit a provider claim for them. Do not include secret values.
Railway says it has closed the exploited routes, but its outstanding-investigation section (opens in new tab) still describes forensic work in progress and public indicators awaiting law-enforcement sign-off. Those findings, and the outcome of customer reimbursement claims, remain the next evidence needed to judge recovery beyond the rebuilt systems.

