Railway disclosed a security incident in which an unauthorized third party may have accessed the encrypted environment variables of its customers and, for some of them, obtained the decrypted contents. Environment variables can hold API keys and database and registry credentials, so Railway recommends that all users rotate every variable and API key stored on the platform as a precaution.

What Railway disclosed

Railway says it detected anomalous activity involving the systems that store customer environment variables. The disclosure, written by Jake Cooper, says initial analysis suggests the impact may be limited to a subset of users, but the rotation advice applies to everyone while the review continues. Railway says it will email affected customers directly once it confirms which accounts were hit.

Railway itself describes the incident this way: "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." Railway says it notified these upstream providers so they could revoke all keys proactively. Railway did not name the providers.

Jake Cooper announced the incident on X at 06:04 on 10/22 Beijing time and recommended that people rotate their keys out of an abundance of caution. Developers who also run model gateways can compare it with the LiteLLM 1.107.0 supply chain compromise, another case where model API keys were the target.

Deployments and the postmortem

All deployments were paused during the investigation, so new versions could not be rolled out. At 12:33 on 10/22 Beijing time, Jake Cooper said Railway had begun ramping builds back up and would continue through the evening, overnight and into the morning. In the same post, Railway said it will publish a full postmortem tomorrow. That post went out on the evening of October 21, Pacific time. Updates are posted on Railway's status page.

What to do now

Railway's guidance, plus two follow-up guides it published for AWS and GCP:

  • Rotate all environment variables stored in Railway right away, including API keys and private registry credentials.
  • Review the logs of the services those keys unlock for unusual activity.
  • AWS keys: deactivate the exposed key, create a replacement, update the variables on Railway and redeploy, then check CloudTrail in every Region and compare Cost Explorer for the exposure window against the previous month.
  • GCP service account keys: disable the exposed key, deploy the new key on Railway, and review Cloud Audit Logs for activity your application did not generate.