I believe the problem was always the process that Google has where they tend to want to automate everything from the start and make it very difficult to reach a human and explain the situation. The service itself seems solid but if you make it difficult to address any problems when(not if) they occur then I won't be comfortable doing business with you. I stay away from their services for anything serious for this reason and always recommend to others to do the same.
It's going to be interesting to watch this unfold. Google's automation vs LLM agents, no humans from either side.
You should publish some of this, as if this is true and AWS and Azure are banning large organisations without realistic recourse on the same scale it should destroy the cloud service providers.
Identified
Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google. The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our Google Cloud–hosted infrastructure and are working to restore the rest of the service. We apologize for the disruption.
Railway had a similar reliability issue two weeks ago when an AI agent deleted a customer's production database via their API — no confirmation step, no environment scoping. Now this. Both incidents suggest the same pattern: infrastructure decisions made without thinking through failure modes, fixed reactively after damage is done.
Agree. The author of that article took 0 responsbility and despite the warnings of "Hey, AI with power in prod is a bad idea" thought "This wouldn't happen to me!" and then when it does "HOW COULD IT DO THIS?!"
So the solution is the same as it has been for over a decade. Don’t do business with Google.
I believe the problem was always the process that Google has where they tend to want to automate everything from the start and make it very difficult to reach a human and explain the situation. The service itself seems solid but if you make it difficult to address any problems when(not if) they occur then I won't be comfortable doing business with you. I stay away from their services for anything serious for this reason and always recommend to others to do the same.
It's going to be interesting to watch this unfold. Google's automation vs LLM agents, no humans from either side.
> Don’t do business with Google.
If you think other big tech or bank won't do it for you if you "don't do anything wrong" - you might be delusional.
The only way to fight it is redundancy. Dont vendor lock in all you stuff in aws/google/cf/whatever. They can (and will) fuck you up at some point.
It is a weekly occurrence for large named organisations and Google.
It isn't for other big tech/banks. It's just a risk assessment, and Google are a high risk partner.
Just because you don't see it, it doesn't meant it doesn't happen.
I deal with it daily. If i would start writing, I would spend half day just writing.
It's in no way to say Google is any good - it's just equally bad as any reasonably big organisation. No more and no less.
You should publish some of this, as if this is true and AWS and Azure are banning large organisations without realistic recourse on the same scale it should destroy the cloud service providers.
Identified Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google. The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our Google Cloud–hosted infrastructure and are working to restore the rest of the service. We apologize for the disruption.
Some discussion started an hour ago (12 points, 3 comments) https://news.ycombinator.com/item?id=48200827
The fix isn't don't use GCP. It's never letting one cloud account suspension take down your control plane and customer workloads simultaneously
Railway had a similar reliability issue two weeks ago when an AI agent deleted a customer's production database via their API — no confirmation step, no environment scoping. Now this. Both incidents suggest the same pattern: infrastructure decisions made without thinking through failure modes, fixed reactively after damage is done.
I wouldn't blame that incident on Railway.. you can delete your prod database on AWS just as easily with their API.
That incident wasn't Railways fault at all. Don't use AI in your staging and prod tools.
Agree. The author of that article took 0 responsbility and despite the warnings of "Hey, AI with power in prod is a bad idea" thought "This wouldn't happen to me!" and then when it does "HOW COULD IT DO THIS?!"
If everyone can delete your prod database via API or by any other means - you need to sack CTO without severance package.