At some point, there aren’t any more tests that you can do, without pushing that system into some kind of production use.
That’s why you do canary deployments. Or blue/green or red/black or whatever color combinations you want to choose.
That’s why you have rollback procedures pre-defined, and tested. Well, as much as they can be tested, until they also get pushed into prod.
That’s why you separate your data plane from your control plane. And you architect your systems so that the data plane can keep running while the control plane is being updated. At least, in some sort of a temporary static condition, while you wait for the control plane to come back.
That’s why you do as much as you can behind load balancers and with automatic failover, preferably in a cross-AZ or cross-region configuration. Or you use any cast mechanisms and let the routers and network switches handle the load balancing and failover for you.
That’s why you use robust algorithms with bounded exponential backoff on retries.
That’s why tools like “chaos monkey” were created.
That’s why you have “game days”, so that you can test all this stuff in the most real-world scenarios that you can engineer.
That’s why you do so many other things to try to keep your systems as robust and bulletproof as possible. And you do this, knowing full well that you can never be perfect at this job. You know that you also need to have a whole quiver full of fast recovery mechanisms you can deploy for those times when the sewage really hits the impellers at a high rate of speed.
At some point, there aren’t any more tests that you can do, without pushing that system into some kind of production use.
That’s why you do canary deployments. Or blue/green or red/black or whatever color combinations you want to choose.
That’s why you have rollback procedures pre-defined, and tested. Well, as much as they can be tested, until they also get pushed into prod.
That’s why you separate your data plane from your control plane. And you architect your systems so that the data plane can keep running while the control plane is being updated. At least, in some sort of a temporary static condition, while you wait for the control plane to come back.
That’s why you do as much as you can behind load balancers and with automatic failover, preferably in a cross-AZ or cross-region configuration. Or you use any cast mechanisms and let the routers and network switches handle the load balancing and failover for you.
That’s why you use robust algorithms with bounded exponential backoff on retries.
That’s why tools like “chaos monkey” were created.
That’s why you have “game days”, so that you can test all this stuff in the most real-world scenarios that you can engineer.
That’s why you do so many other things to try to keep your systems as robust and bulletproof as possible. And you do this, knowing full well that you can never be perfect at this job. You know that you also need to have a whole quiver full of fast recovery mechanisms you can deploy for those times when the sewage really hits the impellers at a high rate of speed.
Yep. There is only prod.