At some point, every growing system hits the "you can't join that anymore" wall. Reads get offloaded to replicas, writes start queueing, and suddenly you're managing backpressure across half a dozen services.
Most teams respond by duct-taping workflows together with message queues, cron jobs, or hand-rolled orchestration logic. It's wild how quickly database scale issues turn into distributed state problems and how rarely people plan for that up front.
I've been leaning into explicit state machines lately (prop for making that less painful). Not because it’s trendy, but because once your system has implicit state transitions buried across services, observability and recovery get brutal.
I've found that once you make the jump to a real partitioned datastore (like DynamoDB) you can actually go back and undo a lot of the queues and caches that were used as band-aids to reduce pressure on the DB. If you have something that has consistent performance at any scale and true elasticity, you just don't need all that other stuff, and the whole system gets far easier to understand and operate.
At some point, every growing system hits the "you can't join that anymore" wall. Reads get offloaded to replicas, writes start queueing, and suddenly you're managing backpressure across half a dozen services.
Most teams respond by duct-taping workflows together with message queues, cron jobs, or hand-rolled orchestration logic. It's wild how quickly database scale issues turn into distributed state problems and how rarely people plan for that up front.
I've been leaning into explicit state machines lately (prop for making that less painful). Not because it’s trendy, but because once your system has implicit state transitions buried across services, observability and recovery get brutal.
I've found that once you make the jump to a real partitioned datastore (like DynamoDB) you can actually go back and undo a lot of the queues and caches that were used as band-aids to reduce pressure on the DB. If you have something that has consistent performance at any scale and true elasticity, you just don't need all that other stuff, and the whole system gets far easier to understand and operate.