This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.
Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.
When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.
In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
This is correct, but there's additional nuance here, which is the security of your development pipeline plays a part in the production system too.
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )
Agreed. I've started calling our non-production infrastructure environments 'engineering' rather than dev, and nobody's invited. We'll pull in platform teams that are exposed to this shame environmental offset issue, but no customers.
In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common.
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
> In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages.
In my experience, we treat "not being able to deploy to prod" as an warning light, because our ability to fix an actual issue or outage in prod is impacted.
i.e. if there is a prod issue, then we can often ether fix it in code, or roll back to an earlier release. But we need the deploy pipeline to be working in order to do that. The longer a pipeline outage persists over time and across multiple services, the more likely it is that there will be some impact.
It's like driving without a seatbelt - you're fine so long as nothing else goes wrong, so it's not recommended as a state to stay in.
> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.
We have a QA team but it’s small in comparison to the dev team. We didn’t have one before as we thought automated tests and “good developers” who take testing seriously was enough. Unfortunately, even with extreme automation and huge test suites, our bug count per release was unacceptable. Having a QA team was a big help in fixing that. We have two weeks of testing before a release goes out. But we sell a security product so for us bugs must be far between. If you just deploy web apps and you can update every hour, I guess you don’t need it as much.
I'm in legacy/finance right now and we have a QA team that we share with maybe 2 or 3 other dev teams. I think our integration testing systems could be a lot more efficient but I think if we had a proper pipeline that truly guaranteed integration testing (right now it feels like a shrug) we would lose the QA team like you did.
Yes. We call them BAs and they have deep product knowledge, so they also get used for training, escalated support requests, estimations on effort required in new integrations, etc.
Here in Finland my company has a dedicated testing team that's almost at large at the actual engineering staff. We sell a combined hardware-software stack, though, and it's basically a contractual requirement in our industry. They are very helpful to be on good terms with when working on a given project, they often know ways to work with the gestalt I never would have guessed in a million years.
Worked for a company with a 3D web application (among other things.)
Those things can't be tested with Playwright et al. So the team had a "every pull request gets an instance like pr143.company.org" pipeline where QA approved. They must still have it.
> Genuine question, does anyone here ITT working in software still have dedicated QA?
A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.
And the whole process improved all across the board.
I think our QAs do pay attention a bit more than that, but some of the things they do flag as issues show (to me) that they don't pay a lot of attention until after we've merged and released or we'd get some of their bugs as requirements on tickets instead of as bugs to resolve between QA deploy and before we go to prod.
Different country probably, here QA is still often part of the team. More often than not as part of a team with a title like "test automation engineer".
My experience is 50/50 - half have dropped them, half still have them.
I also don't find most QA to be worthwhile compared to a good, quality focused dev.
There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.
Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.
Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.
Of course, in these days of vibe coded madness they're probably getting more useful, not less.
The problem with QA is that from the POV of management, all they do is find issues and delay releases. If the QA team did their jobs well, then once all of the reported bugs are found and fixed (passes QA), the software runs fine and there are no future problems. This sounds good in theory, but a good release doesn't get the credit it deserves - it's "business as usual".
A buggy release is fine, if it came out faster, and costed less. The bugs in the release can be fixed in the future, and the "heroic" fixes are "creditworthy".
Therefore, over time, the QA team is politically relegated as a "useless" team in the department, and thus given how politics play out, they get sidelined until the entire team is gone.
This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)
When we do updates, I mandate that our teams do blue/green deployments meaning they setup the exact VM side by side with the actual production data and test with the new version of software (eg GitLab) before doing the DNS flip. This is on prem. It’s an ephemeral sandbox.
I don’t think I’ve worked in an industry where it’s safe to use production data in a non-prod environment in probably 20 years.
I think for dev, it’s fine not to have blue/green. You want dev deployments to be fast. UAT / pre-production/ or whatever your final env before prod — and particularly if stakeholders are intended to test on that env — is a different matter.
You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment.
Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR.
This isn’t only for your (deployed) Production environment.
It’s also not software development 101. This has roots in ITIL.
> You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment.
You’re the one drifting all over the place with the terminology.
I get what you’re trying to say. But you’re saying it badly.
You have your dev, test, UAT, production, etc environments. They’re the “software development” envs as you named them.
You then have your DevOps envs. You call them sandbox, but Personally I hate that term because the DevOps envs should be the same as the “software development” envs just not accessible to the software developers. A “pre-dev” env if you will. So it should be the same IaC and the same build pipelines.
Whereas a “sandbox” generally suggests that it’s “ClickOps” / manually provisioned envs with no rigour. A place to
play and experiment before you’re ready to commit your infra to code. Which is not how the DevOps envs should exist. The DevOps pre-dev envs should still be a carbon copy of the “software developer” environments.
> ithin your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR
No it shouldn’t. HA, DR, etc costs money. Blue/green is slow and you shouldn’t let DevOps be a blocker for development feedback loops. So you save that stuff for your pre-production envs, and keep your dev and test envs nimble and quick to deploy. The DR for your lowest environments should just be “destroy and recreate” because you’ll have pipelines to deploy that infra.
In short. You can still have the same approximate infra but you shouldn’t emulate production best practices precisely across all development environments.
If you find you cannot have a “production-like” dev but without the HA, DR, etc, then you have designed your pipelines wrong from the outset. Eg You’ve got “pets instead of cattle” (ie the lack of ephemeral infra), manually provisioned services, and other faux pas.
> This isn’t only for your (deployed) Production environment
It isn’t. But it’s also not for all environments indiscriminately either.
> It’s also not software development 101. This has roots in ITIL.
Of course it’s software development 101. If your developers don’t understand the basics of different software development environments then they’re either shit at their job or fresh out of uni. I don’t mean to be harsh here but I don’t know how else to emphasise that what you’re describing here is the absolute basics. 30 years ago, when I was starting out, I learned this before I learned about version control. That’s how basic a topic this is.
Yeah at my place our DevOps team had been asking for resources on our CI/CD pipeline for a long time but wasn't prioritized. After awhile, after introducing something like teams wide "Retros', we noticed they didnt want the resources, we wanted them!
So we started being more vocal and management understood its for everyone and their wish got approved recently and its been a big change.
There's more to making an argument than stating your conclusion forcefully and repeatedly.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
Not seeing this get traction, but this is exactly what I thought while reading this. The problem with calling something "a" top priority is you can only have one top priority. That's how priority works. A customer facing outage is a higher priority than a company-wide developer tool outage, which is higher priority than a single developer's personal workstation failing, etc. etc. That doesn't mean the lower-priority things don't matter, but you can't make any and every failure of anything an all hands on deck situation. That's a certain path to burning out your entire company.
True. So you're talking about allocating resources (teams) across priorities. Which have to be prioritized. Which is the reality the article doesn't wrestle with at all.
It doesn’t have to because the premise of the article is that development environments akin to manufacturing (eg software factories) need to be treated like Production. It’s a given that you have enough resources to handle both.
So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.
That sounds nice and pleasantly confirms our biases, but it's blatantly false.
Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.
If anything, recruitment in our industry was always broken. And yet here we are.
You aren’t wrong. That’s why recruiters and placement agencies exist. The statement is still true though. It’s just mostly deferred now onto another entity.
No it's not. If the development pipeline breaks on Saturday night there is absolutely no need to go fix it right away.
You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while important, is not production)
I feel like your example exactly describes why the development pipeline is production. Just because the critical hours of the system are different doesn't mean it's any less important.
I’d bet there are as many stories of businesses failing because of inability to ship quickly as there are about focusing too much on your tooling instead of delivering value to customers. Both failures are dangerous! And depending on your market, product, team etc. a different spot on that spectrum is appropriate.
In some places shipping quickly is part of the value. In others, the product works and delivers value and having slow releases (implying eg a manual release process) is a feature.
What this article advocates for is certainly a valid lens, by IMO it’d be a mistake to take it as universal.
I worked at a startup that was briefly a unicorn and ended up burning cash because they did exactly what you said of focusing a lot on tooling and dev environment. It was great but way over the top for what we needed. I learned a lot of good practices for myself but they've been limping along for a while and most of my former team was laid off or saw what was coming and left before it hit them.
Worth mentioning CI is not part of production outages!
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
Having spent too many years in CI/CD I'd agree but there is an even better solution; decouple deployments from releases. Make it possible to deploy without releasing, and release without deploying. (An implementation of this could be, for example: feature flags.)
I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem?
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
> how do you properly measure it?
> If a problem never manifests, but was prevented, was it really a problem?
For the scheduled half of it, i like inverting the alert. Instead of alerting when a job fails or wait for data leak, have the job check in on every successful run and alert when a check-in doesn't arrive. Dead man's switch. Ten minutes to set up, and it's the difference between "the cron is fine" and a statement that can actually turn out to be false.
Agree that it is difficult to implement for any process, but it is easier to create validation systems than "find the error" processes.
Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time.
Some packages may also drop support for whatever distro you're still using, and the mirrors may disappear (e.g. older Python versions, ROS, etc.).
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you.
To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.
Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.
Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.
This is why some projects will vendor their dependencies - changes within the dependency itself will also be reviewed, and it's easier to locally maintain patches to those dependencies until the patches can be upstreamed.
The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosystem dependent.
The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.
That works for artifacts, but there's no easy solutions for all the security tools that relies on pulling up to date external databases. Databases that are out of your control and often down in my experience.
Of course, anything deployed for use by the target customer is a production system. And anything that is under development is a dev system. Pipeline itself is not under development and it is in use by the target customer (developer). What's the confusion here?
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
Recently I heard a term "Software factory" - I am strongly rooting with this term, particularly while working with agents and AI. Heard from Vercel, Guillermo.
"The (software) factory is the product. Your product is only as good as the agents you set up to autonomously maintain it." https://x.com/rauchg/status/2081123293340520642?s=20
Mechanized production was designed by people to produce commodities and was designed with planned tradeoffs, tolerances, and failure modes. That’s not what agents are, though I get why an analogy to the Industrial Revolution is useful for CEOs.
I thought this was going to be about AI / agentic development stacks.
Because long running, well tuned pipelines can deliver on some important goals but be broken by the changes from adopting a new model (ahem opus 5) or rot in some key web automation path.
The failure mode I hit today isn't CI running untrusted code — it's simpler and I suspect more common. The deploy target contained state that wasn't in the source tree.
Deploy was an rsync of a few directories to a server. That server directory also held .env.local, and a content/ dir that a scheduled job writes to directly. Neither is in git. Someone (me) added --delete so the destination would "match source", and rsync faithfully removed everything outside the source set: the env file, the five .bak copies sitting next to it, and a week of generated content.
Two accidents kept it from being worse. The rsync excluded the build output, so the already-running process kept serving; and because the app had read its secrets into memory at boot, checkout kept working off a file that no longer existed on disk. That bought a few hours to restore from git.
What I'd add to the article: treating the pipeline as production should include auditing the pipeline's write scope. Mine had unrestricted write+delete over a directory holding live credentials, and that was invisible until it mattered. Also — backups in the same directory as the original are not backups.
There is a weird duality when analyzing this... On the one hand, I think factory assembly lines are just a high bar to clear as as metaphor for internal processes. Almost no business actually runs a super organized backend workflow; there is always "I thought you followed up with the customer" "I thought you did"
On the other hand if the email server is down that's just as big a problem for any biz as if customer facing product was not working so some workflow things are already treated as production tier priority
this muddying of terms is not useful. No, my IDE breaking down is not a production system failure. It's a development system failure and sure, that can very easily have consequences on the production system but usually not immediately. It should be obvious to everyone how having a functional hammer is going to be important if you want to fix a house. There is no need to start calling the hammer the house.
One of my mantras for decades has been: make change easy. Once you invest in great CI/CD, you can change stuff quickly and reliably. Then you can easily fix/roll back production issues. The dev team doesn't fear production deployments once change is easy.
Yes but parts of it must live in the dev environment. CI by its very nature runs untrusted code. That is very hard to secure. Keep it away from production, production artifacts and production credentials. CI and CD should also be separate systems. The same system which builds and runs untrusted code should not have permission to push to production. Most of the industry uses a single system for CI/CD though. Good luck to you all.
At my (now former, thanks layoffs) company, we used to gripe about getting nontechnical management to understand how stupid it was to skimp on pre-prod funding . . . because pre-prod was our prod.
You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on.
And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd.
Tldr? Go ahead and page yourself for non-prod alerts. No one will stop you.
I wouldn't begrudge an actual human having an issue at 2am. But a wave of alerts for servers that were retired but their alerting wasn't, or hair trigger thresholds that pop when a new crown job pushes the load up 1% over WARN...
The issue is "production is defined by what humans are expected to maintain all night long" and dev ain't prod broskies.
Being a production system doesn't make it a priority.
> A team with a broken development pipeline can’t produce software, and must treat this as a production outage.
While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing.
To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes.
Increasingly though founders and executive team are non-technical and simply don't care until it shows up on a graph that piques their interest (the ones with $ somewhere).
This is a great way to encourage your devs to jump ship as soon as anything even a little better comes up.
It's trendy to act like devs are fungible resources or that AI will replace them, but the truth is that hiring smart people, paying them lots of money, and then forcing them to spend half their time dealing with prod issues that they're not empowered to fix is a staggering waste of resources.
Since deployment or testing issues are mostly invisible to upper management (who never have to spend half of THEIR time fixing them) this doesn't come with any recognition or promotions, so the only thing left to absorb the impact is morale and goodwill towards the company.
Sadly, engineer-driven companies are increasingly rare.
This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
For 26 years I have tried and failed to explain this to my different managers at so many different jobs. I gave up.
I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.
Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
At some point, it becomes hard to assert “we are NOT going to test the product the way a customer would actually use it”
Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.
We can do that on our developers' machines no problem, but there still needs to be a place for QA to do their testing.
Can QA not do the same thing?
When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.
In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
This is correct, but there's additional nuance here, which is the security of your development pipeline plays a part in the production system too.
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )
Agreed. I've started calling our non-production infrastructure environments 'engineering' rather than dev, and nobody's invited. We'll pull in platform teams that are exposed to this shame environmental offset issue, but no customers.
In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages. Being on-call in a CI/CD infra team is fairly common.
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
It's bigger than that even. Ensuring that the code you are shipping has not been tampered with, for example.
> In my experience, most large companies do treat not being able to ship code (i.e. deploy to prod) as outages.
In my experience, we treat "not being able to deploy to prod" as an warning light, because our ability to fix an actual issue or outage in prod is impacted.
i.e. if there is a prod issue, then we can often ether fix it in code, or roll back to an earlier release. But we need the deploy pipeline to be working in order to do that. The longer a pipeline outage persists over time and across multiple services, the more likely it is that there will be some impact.
It's like driving without a seatbelt - you're fine so long as nothing else goes wrong, so it's not recommended as a state to stay in.
> If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.
We have a QA team but it’s small in comparison to the dev team. We didn’t have one before as we thought automated tests and “good developers” who take testing seriously was enough. Unfortunately, even with extreme automation and huge test suites, our bug count per release was unacceptable. Having a QA team was a big help in fixing that. We have two weeks of testing before a release goes out. But we sell a security product so for us bugs must be far between. If you just deploy web apps and you can update every hour, I guess you don’t need it as much.
I'm in legacy/finance right now and we have a QA team that we share with maybe 2 or 3 other dev teams. I think our integration testing systems could be a lot more efficient but I think if we had a proper pipeline that truly guaranteed integration testing (right now it feels like a shrug) we would lose the QA team like you did.
Yes. We call them BAs and they have deep product knowledge, so they also get used for training, escalated support requests, estimations on effort required in new integrations, etc.
Here in Finland my company has a dedicated testing team that's almost at large at the actual engineering staff. We sell a combined hardware-software stack, though, and it's basically a contractual requirement in our industry. They are very helpful to be on good terms with when working on a given project, they often know ways to work with the gestalt I never would have guessed in a million years.
Worked for a company with a 3D web application (among other things.)
Those things can't be tested with Playwright et al. So the team had a "every pull request gets an instance like pr143.company.org" pipeline where QA approved. They must still have it.
The other companies tested "on demand."
> Genuine question, does anyone here ITT working in software still have dedicated QA?
A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.
And the whole process improved all across the board.
I think our QAs do pay attention a bit more than that, but some of the things they do flag as issues show (to me) that they don't pay a lot of attention until after we've merged and released or we'd get some of their bugs as requirements on tickets instead of as bugs to resolve between QA deploy and before we go to prod.
Different country probably, here QA is still often part of the team. More often than not as part of a team with a title like "test automation engineer".
We have 2 QA and I’m grateful every day for them. The devs generate slop at light speed but nothing gets passed those guys, they’re fantastic.
Oh exactly the same here.
Small company in Germany, QA team with about 5 people. But not all the devs are bad, just some are too comfortable with AI and getting lazy.
"past"
Yes. I work in games and we heavily rely on manual QA. Too heavily, really.
My experience is 50/50 - half have dropped them, half still have them.
I also don't find most QA to be worthwhile compared to a good, quality focused dev.
There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.
Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.
Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.
Of course, in these days of vibe coded madness they're probably getting more useful, not less.
The problem with QA is that from the POV of management, all they do is find issues and delay releases. If the QA team did their jobs well, then once all of the reported bugs are found and fixed (passes QA), the software runs fine and there are no future problems. This sounds good in theory, but a good release doesn't get the credit it deserves - it's "business as usual".
A buggy release is fine, if it came out faster, and costed less. The bugs in the release can be fixed in the future, and the "heroic" fixes are "creditworthy".
Therefore, over time, the QA team is politically relegated as a "useless" team in the department, and thus given how politics play out, they get sidelined until the entire team is gone.
This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)
When we do updates, I mandate that our teams do blue/green deployments meaning they setup the exact VM side by side with the actual production data and test with the new version of software (eg GitLab) before doing the DNS flip. This is on prem. It’s an ephemeral sandbox.
I don’t think I’ve worked in an industry where it’s safe to use production data in a non-prod environment in probably 20 years.
I think for dev, it’s fine not to have blue/green. You want dev deployments to be fast. UAT / pre-production/ or whatever your final env before prod — and particularly if stakeholders are intended to test on that env — is a different matter.
When I say Production, I mean Production Software Development data like Coverity or Fortify databases. GitLab source code and pipelines.
The premise is that Software Development is akin to Production which it should be.
That’s not the same thing as what people generally mean when they say “production data”.
And yes, your dev environments should mirror the production environments bar production data. That’s software development 101.
You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment.
Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR.
This isn’t only for your (deployed) Production environment.
It’s also not software development 101. This has roots in ITIL.
> You are still mixing production and dev contexts. What is called software development environment is production for your software factory. It’s not development in terms of a sandbox for your deployment, Production environment.
You’re the one drifting all over the place with the terminology.
I get what you’re trying to say. But you’re saying it badly.
You have your dev, test, UAT, production, etc environments. They’re the “software development” envs as you named them.
You then have your DevOps envs. You call them sandbox, but Personally I hate that term because the DevOps envs should be the same as the “software development” envs just not accessible to the software developers. A “pre-dev” env if you will. So it should be the same IaC and the same build pipelines.
Whereas a “sandbox” generally suggests that it’s “ClickOps” / manually provisioned envs with no rigour. A place to play and experiment before you’re ready to commit your infra to code. Which is not how the DevOps envs should exist. The DevOps pre-dev envs should still be a carbon copy of the “software developer” environments.
> ithin your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR
No it shouldn’t. HA, DR, etc costs money. Blue/green is slow and you shouldn’t let DevOps be a blocker for development feedback loops. So you save that stuff for your pre-production envs, and keep your dev and test envs nimble and quick to deploy. The DR for your lowest environments should just be “destroy and recreate” because you’ll have pipelines to deploy that infra.
In short. You can still have the same approximate infra but you shouldn’t emulate production best practices precisely across all development environments.
If you find you cannot have a “production-like” dev but without the HA, DR, etc, then you have designed your pipelines wrong from the outset. Eg You’ve got “pets instead of cattle” (ie the lack of ephemeral infra), manually provisioned services, and other faux pas.
> This isn’t only for your (deployed) Production environment
It isn’t. But it’s also not for all environments indiscriminately either.
> It’s also not software development 101. This has roots in ITIL.
Of course it’s software development 101. If your developers don’t understand the basics of different software development environments then they’re either shit at their job or fresh out of uni. I don’t mean to be harsh here but I don’t know how else to emphasise that what you’re describing here is the absolute basics. 30 years ago, when I was starting out, I learned this before I learned about version control. That’s how basic a topic this is.
Yeah at my place our DevOps team had been asking for resources on our CI/CD pipeline for a long time but wasn't prioritized. After awhile, after introducing something like teams wide "Retros', we noticed they didnt want the resources, we wanted them!
So we started being more vocal and management understood its for everyone and their wish got approved recently and its been a big change.
There's more to making an argument than stating your conclusion forcefully and repeatedly.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
Not seeing this get traction, but this is exactly what I thought while reading this. The problem with calling something "a" top priority is you can only have one top priority. That's how priority works. A customer facing outage is a higher priority than a company-wide developer tool outage, which is higher priority than a single developer's personal workstation failing, etc. etc. That doesn't mean the lower-priority things don't matter, but you can't make any and every failure of anything an all hands on deck situation. That's a certain path to burning out your entire company.
You need more than one team working outages. It’s not one team doing it all.
True. So you're talking about allocating resources (teams) across priorities. Which have to be prioritized. Which is the reality the article doesn't wrestle with at all.
It doesn’t have to because the premise of the article is that development environments akin to manufacturing (eg software factories) need to be treated like Production. It’s a given that you have enough resources to handle both.
So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.
That sounds nice and pleasantly confirms our biases, but it's blatantly false.
Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.
If anything, recruitment in our industry was always broken. And yet here we are.
You aren’t wrong. That’s why recruiters and placement agencies exist. The statement is still true though. It’s just mostly deferred now onto another entity.
No it's not. If the development pipeline breaks on Saturday night there is absolutely no need to go fix it right away.
You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while important, is not production)
I feel like your example exactly describes why the development pipeline is production. Just because the critical hours of the system are different doesn't mean it's any less important.
I have never worked in a company where it is okay for production to be down at any time. Critical hours or not.
And if production is down even if there is just one user, it is all hands on deck.
I’d bet there are as many stories of businesses failing because of inability to ship quickly as there are about focusing too much on your tooling instead of delivering value to customers. Both failures are dangerous! And depending on your market, product, team etc. a different spot on that spectrum is appropriate.
In some places shipping quickly is part of the value. In others, the product works and delivers value and having slow releases (implying eg a manual release process) is a feature.
What this article advocates for is certainly a valid lens, by IMO it’d be a mistake to take it as universal.
I worked at a startup that was briefly a unicorn and ended up burning cash because they did exactly what you said of focusing a lot on tooling and dev environment. It was great but way over the top for what we needed. I learned a lot of good practices for myself but they've been limping along for a while and most of my former team was laid off or saw what was coming and left before it hit them.
Worth mentioning CI is not part of production outages!
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
Having spent too many years in CI/CD I'd agree but there is an even better solution; decouple deployments from releases. Make it possible to deploy without releasing, and release without deploying. (An implementation of this could be, for example: feature flags.)
I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem?
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
> how do you properly measure it? > If a problem never manifests, but was prevented, was it really a problem?
For the scheduled half of it, i like inverting the alert. Instead of alerting when a job fails or wait for data leak, have the job check in on every successful run and alert when a check-in doesn't arrive. Dead man's switch. Ten minutes to set up, and it's the difference between "the cron is fine" and a statement that can actually turn out to be false.
Agree that it is difficult to implement for any process, but it is easier to create validation systems than "find the error" processes.
https://dora.dev/guides/dora-metrics/
Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're still using, and the mirrors may disappear (e.g. older Python versions, ROS, etc.).
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you.
To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.
Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.
> to be in control of the dependencies
Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.
This is why some projects will vendor their dependencies - changes within the dependency itself will also be reviewed, and it's easier to locally maintain patches to those dependencies until the patches can be upstreamed.
The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosystem dependent.
The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.
That works for artifacts, but there's no easy solutions for all the security tools that relies on pulling up to date external databases. Databases that are out of your control and often down in my experience.
I think I disagree
There is a production eco-system - let’s call it The Hotel. It runs, people work in it and customers pay and it makes money.
But it constantly needs changing. New rooms need to be built, the kitchen rearranged as the chef changes his mind.
So they hire an architect and building firm. Who plan the chnages and then this awesome automated robot comes and builds the extra room.
The architect firm owns and is responsible for the robot and its changes.
Not being able to make a change really really matters, but I don’t call that production because the hotel is still able to operate
Most of the misunderstanding between business and tech is a misunderstanding of are you working on the architect firm or the hotel
Of course, anything deployed for use by the target customer is a production system. And anything that is under development is a dev system. Pipeline itself is not under development and it is in use by the target customer (developer). What's the confusion here?
> What's the confusion here?
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
Recently I heard a term "Software factory" - I am strongly rooting with this term, particularly while working with agents and AI. Heard from Vercel, Guillermo. "The (software) factory is the product. Your product is only as good as the agents you set up to autonomously maintain it." https://x.com/rauchg/status/2081123293340520642?s=20
Mechanized production was designed by people to produce commodities and was designed with planned tradeoffs, tolerances, and failure modes. That’s not what agents are, though I get why an analogy to the Industrial Revolution is useful for CEOs.
I thought this was going to be about AI / agentic development stacks.
Because long running, well tuned pipelines can deliver on some important goals but be broken by the changes from adopting a new model (ahem opus 5) or rot in some key web automation path.
The failure mode I hit today isn't CI running untrusted code — it's simpler and I suspect more common. The deploy target contained state that wasn't in the source tree.
Deploy was an rsync of a few directories to a server. That server directory also held .env.local, and a content/ dir that a scheduled job writes to directly. Neither is in git. Someone (me) added --delete so the destination would "match source", and rsync faithfully removed everything outside the source set: the env file, the five .bak copies sitting next to it, and a week of generated content.
Two accidents kept it from being worse. The rsync excluded the build output, so the already-running process kept serving; and because the app had read its secrets into memory at boot, checkout kept working off a file that no longer existed on disk. That bought a few hours to restore from git.
What I'd add to the article: treating the pipeline as production should include auditing the pipeline's write scope. Mine had unrestricted write+delete over a directory holding live credentials, and that was invisible until it mattered. Also — backups in the same directory as the original are not backups.
There is a weird duality when analyzing this... On the one hand, I think factory assembly lines are just a high bar to clear as as metaphor for internal processes. Almost no business actually runs a super organized backend workflow; there is always "I thought you followed up with the customer" "I thought you did"
On the other hand if the email server is down that's just as big a problem for any biz as if customer facing product was not working so some workflow things are already treated as production tier priority
this muddying of terms is not useful. No, my IDE breaking down is not a production system failure. It's a development system failure and sure, that can very easily have consequences on the production system but usually not immediately. It should be obvious to everyone how having a functional hammer is going to be important if you want to fix a house. There is no need to start calling the hammer the house.
One of my mantras for decades has been: make change easy. Once you invest in great CI/CD, you can change stuff quickly and reliably. Then you can easily fix/roll back production issues. The dev team doesn't fear production deployments once change is easy.
From the article title, I was hoping that the author would argue that software development could be modeled as one of these. https://en.wikipedia.org/wiki/Production_system_(computer_sc...
This is a highly accurate and unappreciated take.
There’s the system you are building…and the system that builds the system or the SDLC. Both are critical.
Solid read.
Yes but parts of it must live in the dev environment. CI by its very nature runs untrusted code. That is very hard to secure. Keep it away from production, production artifacts and production credentials. CI and CD should also be separate systems. The same system which builds and runs untrusted code should not have permission to push to production. Most of the industry uses a single system for CI/CD though. Good luck to you all.
Clear, useful, done
At my (now former, thanks layoffs) company, we used to gripe about getting nontechnical management to understand how stupid it was to skimp on pre-prod funding . . . because pre-prod was our prod.
no shit. another insightless article.
Sounds like someone wants to go on-call.
You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on.
And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd.
Tldr? Go ahead and page yourself for non-prod alerts. No one will stop you.
2:34 AM
zZzzZ...
2:35 AM
ALERT #42 - CRITICAL
!!!
ALERT #42 - CRITICAL - Joan forgot feir password to npm
Help feir!
I wouldn't begrudge an actual human having an issue at 2am. But a wave of alerts for servers that were retired but their alerting wasn't, or hair trigger thresholds that pop when a new crown job pushes the load up 1% over WARN...
The issue is "production is defined by what humans are expected to maintain all night long" and dev ain't prod broskies.
Being a production system doesn't make it a priority.
> A team with a broken development pipeline can’t produce software, and must treat this as a production outage.
While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing.
To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes.
Increasingly though founders and executive team are non-technical and simply don't care until it shows up on a graph that piques their interest (the ones with $ somewhere).
This is a great way to encourage your devs to jump ship as soon as anything even a little better comes up.
It's trendy to act like devs are fungible resources or that AI will replace them, but the truth is that hiring smart people, paying them lots of money, and then forcing them to spend half their time dealing with prod issues that they're not empowered to fix is a staggering waste of resources.
Since deployment or testing issues are mostly invisible to upper management (who never have to spend half of THEIR time fixing them) this doesn't come with any recognition or promotions, so the only thing left to absorb the impact is morale and goodwill towards the company.
Sadly, engineer-driven companies are increasingly rare.