Fun fact: The first-ever Durable Execution POC at AWS Simple Workflow used snapshots. The workflow was implemented as a fully asynchronous Java application, and a snapshot was a dump of the whole object graph using reflection. Performance was abysmal because a snapshot had to be generated after every state transition.
So we switched to replay. The biggest benefit of replay is that it lets you implement Durable Execution in any language as a library without a complex runtime. It also supports code changes while workflows are in flight (Temporal calls this patching). Making snapshots of arbitrary code state backward-compatible with code changes isn't practical.
I personally think that, in the long term, Durable Execution will use a runtime that supports both snapshotting and determinism. That way, snapshots can be taken infrequently, and replay can bring workflow code to the latest state. Similarly to a database recovering from a WAL. WASM is the most promising technology to achieve this.
I've built a framework that provides durable threads using serializable continuations in Java (with a modified JVM) and all these issues are easily solvable. Sadly it's not public :( The framework actually does have solutions for those, and all the issues raised in the article, and does support hot patching too. Plus it has a nice waitUntil() API that lets you sleep until an arbitrary combination of events including database query changes.
Performance versus log replay depends a lot on what you're doing, there are plenty of cases where snapshots are faster. Consider anything where you download a lot of data and filter it. But I argue the programming model of log replay is so terrible, and creates so many new classes of subtle bugs, that it's worth paying almost any performance price to get away from it. Especially for durable workflows correctness matters more than performance and it's much easier to achieve with continuation.
In the end it wasn't necessary (because workflows aren't expected to be super fast) but if needed I could have optimized snapshotting further with some more JVM changes. The JVM I was working with is written in Java so is easy to modify.
For hot patching there are a few tricks that help.
1. Only store live data. If a variable points to a large object graph before a checkpoint but isn't used afterwards, don't snapshot it.
2. Make it easy to switch the version of a running continuation only at known-safe checkpoints. I identify checkpoints with a (stack trace, counter) pair.
3. Mostly people want hotpatching only at specific points in their program, typically at the top of an infinite loop that's waiting for something. Design the scheduler so you can expose an API that offers "wait until something happens that I'm interested in, or I change version and then hot swap me", with a test framework that actually drives continuations through those sorts of hotswaps. If you get the API right then you (framework author) control what's live on the stack at that moment and the developer just has to think about the core state of their main root object, which they'd need to think about anyway and is where the important stuff is.
This is better than hotswap/patching with log replay, which is extremely risky - you can't change code at a given point even if you know all your workflows are beyond that point, so it's a leaky abstraction. And you just can't upgrade infinite loops at all, which makes hotswap a lot less useful to begin with.
Good model. Anybody interested in actually training models or designing agentic systems should be doing this.
My company started around working on this problem because it's the basis for how you train programming models/reliably deploy LLMs to do specific tasks. It allowed me to build a much better mental model for LLMs because I saw how weirdly fickle/inconsistent/picky they could actually be outside of a "chat" where it feels like they have a coherent persona or consistent knowledge/capability.
Initially I thought of it as a search over prompts for capability at completing specific tasks, but now I think the speed/reliability and operations (eg can I switch models without degrading perforamnce?) benefits are even bigger benefits for most users.
A little "secret" since labs are making it harder to even use their models in this way and it's important that it be more widely understood: distribution-aware replay/re-sampling is a key technique in post-training LLMs. But it's also something that allows you to automatically identify the best model for some subset of your tasks, which can save you a lot of money.
Durable execution looks a bit like a buzzword in general. If your state is defined in the execution graph then it's just an umbrella term for a group of pre-existing algorithms and patterns. If it's undefined then what are you resuming to? The snapshot just before the crash likely leads to the undefined state again, in which case you're durably automating the crash (or even worse, uncaught incorrect behavior).
> Durable execution looks a bit like a buzzword in general. If your state is defined in the execution graph then it's just an umbrella term for a group of pre-existing algorithms and patterns.
If you first approach a tool because of buzzwords, don't be surprised that you think of the buzzwords instead of the tool.
Durable executions greatly simplify how workflows can be implemented and audited, and they literally allow eternal workflow executions that are not tied to the lifetime of an instance assigned to execute them. Durable executions do not require fancy infrastructure, only a terribly simple database.
If you understand continuations, you understand durable executions. Otherwise, you'll be stuck with the excuse that they are buzzwords.
Well, you have to either capture/eliminate/persist side effects and control the environment tightly, or it's limited in what it can do.
In a distributed or concurrent system, for full granularity, that can require specialized timing or virtualization techniques up to ensuring fully atomic snapshots and deterministic execution environments (and whether or not that properly models the SUT in real environments, or introduces bias/breaks the reproducibility in a way you care about)
Otherwise if you're only running against fixed checkpoints you have something closer to traces that maybe you could re-run or test against, in some cases, if you put in the work to set it up. In distributed systems that can be a lot of work so it's a bit vague if left unspecified. Because it's not enough to merely replay something if things can drift or don't accurately model the real system
> Well, you have to either capture/eliminate/persist side effects and control the environment tightly, or it's limited in what it can do.
You really need to read up on durable executions before commenting. Your comment reads as if you are completely oblivious to them. Their whole point is benefiting from an execution model where your workflow is comprised of idempotent pure functions whose inputs and outputs are tracked by the durable task persistence.
It's a problem I've spent a lot of time on, just not through the products marketed like temporal. I commented on the article because it's about program replay with checkpointing which is similar to what I worked with/mentioned.
I didn't see the article mention idempotency anywhere, and you didn't in the original response to the guy who said it just sounded like a buzzword; atomic snapshotting with deterministic execution is literally how you make a program continuation an idempotent function!
And the article is about solving the problem at the language runtime level so it doesn't even do that. So it would be reasonable to assume it is a buzzword if it does not have the essential property you mentioned and I was referring to. I was not even intending to disagree with you but just add what would make it less of a buzzword in this kind of case.
For the sort of architecture described in the blog post, you're persisting the program rather than a log, so you only need idempotency between any two checkpoints. For a lot of apps that's easy to achieve if write the snapshot into the same transaction the user is using for application data, and if you create/propagate idempotency tokens to other services.
You don't need specialized timing or virtualization techniques.
Well checkpointing does not make an incorrect program correct. Durability just guarantees that execution can survive infrastructure failure, not that the resumed program will not encounter the same deterministic bug again.
TCC checkpoints at explicit durable boundaries rather than at an arbitrary instruction immediately before a crash. So if the next operation repeatedly fails, ordinary retry limits or operator intervention are still required.
My proposed model is to represent the recoverable program state as a committed continuation, rather than reconstructing state by replaying the completed execution history.
Our architecture has had somewhat of a different primitive for years that yields the same outcome.
Our application architecture is named The Feature Architecture.
A unit of work is feature. A lot of common implementation can be derived from (or compressed into) the name of a feature. They are invoked by Features.invoke(feature_name,...) and seamlessly calls same service, service to service or frontend to backend service or even backend to frontend (with client ID and userid) seamlessly.
A command feature by default does durable execution by being a consumer as well as a feature as the machinery ensures all incoming command feature messages are injected into monolog (in house built akin to kafka). So command feature errors are retried by the machinery by default.
All Dip operations are idempotent and hence can be retried maximally.
Our arcc (The architecture compiler) enforces at compile time that
1) there are zero CQRS violations of query to command invocations
2) zero violations of query to Dip.insert/update/remove (extended CQRS for persistence)
3) all Dip mutate operations are idempotent (arcc --strict),
4) zero alien Dip collection access (a feature leaf and its handlers owns exactly one collection)
5) and a whole myriad of around 10 different architecture rules that usually depend on developer discipline and conventions.
One can set a command operation to be not a durable execution by specifying bypass: true for that feature in the registry but those are the outliers.
Checkpointing and replay are not mutually exclusive in our architecture as they emerge when a command feature is either a default or one with bypass: true.
So durable executions are inherently native and first class for all commands in our Feature Architecture.
Yes, Monty is highly relevant. Its ability to serialize and resume a paused interpreter has a clear relationship to this work. I would say the main difference is scope; Monty is a secure Python interpreter for AI-written code, while TCC is exploring durable execution semantics around effects, waits, child executions, and persisted continuations, eventually across language frontends. Thanks for the insight, I'll definitely add Monty to the related-work discussion.
I’ve been building a durable execution framework with roughly the same mechanism. It’s a pretty logical rough edge to try shave off from Temporal-like systems.
Good question. I operate a pretty big Temporal deployment at work (~3.45 billion actions per month), and also have the dubious distinction of being one of the only community SDK (Haskell) implementors for Temporal. So I have a pretty good sense of rough edges and my own areas of dissatisfaction, and also what I'd want out of a next generation system.
Which rough edges pushed you toward the same mechanism - history growth, replay latency, determinism/versioning, worker ops, or the programming model? Happy to compare notes.
Definitely friends, not frenemies. I have nothing but genuine respect for builders out there solving hard infrastructure problems.
We are primarily focused on our Feature Architecture. Durable execution is only for command features, requires tight coupling with our architecture, and is not a standalone dedicated tool like Temporal.
Open to further discussion on both of our work's trajectories rather than digress here.
Building for extreme scale constrained the possible architectural space. For a high level overview of the application architecture, refer to the seperate comment here in this thread.
It started initially by realizing that our app, Slyp, requires extremely distributed invoice and coupon processing as we started rolling out. So the architecture must handle such scale without issues.
This lead to persisting every incoming request (for commands) from the frontend and dismissing them only to be informed later of continuing with the status. This avoids processing requests when the system is overloaded with high traffic. So naturally the minimum is a log style ordered persistence. Initially kafka. Later our own Monolog built in rust which is substantially faster for our workloads and zero jvm and low memory and can run on servers and mobile alike.
Naturally the right point to consume this was into the command ingestion point in the Feature Architecture as noted in the other comment. This expanded the capability to all feature invocations including inter and intra service feature invocations.
This then progressively evolved into a much capable engine at which point we realized, this is a durable execution engine for command features but as a minor part of the whole Feature Architecture itself.
Was this written by AI? I even read the "technical paper" and still don't understand what it's supposed to do. Nowhere does it explain how this "checkpointing" and "resuming a continuation" actually works in practice.
Fair point. The paper explains the execution model and evaluation, but I should have included a concrete source-to-continuation transformation. I kept the implementation discussion too abstract while the design was still evolving.
Essentially, the compiler lowers the program into an explicit state machine, and a committed checkpoint contains the next state identifier along with live values and any relevant control states needed to resume execution. For recovery, a fresh runtime hydrates that frame and resumes directly from the saved state rather than replaying the completed history.
Checkpointing the live continuation shifts recovery cost toward retained state rather than lifetime event count, which seems especially useful for agents with many completed tool calls.
Yup, essentially capture state now so execution can continue from it later. The difference is that this checkpoint represents compiler-known program state (live values and control position) rather than an entire process or address space. That lets it omit dead state and the completed execution prefix.
dropping history replay would remove so much operational pain from durable workflows. how do you reconstruct in-flight state after a crash, snapshotting or something event sourced
Closer to compiler-generated continuation checkpointing than event sourcing.
In the current prototype, the compiler lowers the program into an explicit state machine, and then at each durable boundary - waits, external effects, child executions, etc - it commits the next program position along with live locals, control state, and any values needed to continue execution.
When the program crashes or is killed while executing, a fresh runtime loads the checkpoint, reconstructs the frame, and dispatches directly to the saved position - no history replay.
External effects still have the usual ambiguity window: an operation may succeed externally, but the process may crash before its result and the updated checkpoint are durably committed. To solve this, TCC gives each effect a stable identity, but non-idempotent operations still require provider-supported idempotency or reconciliation. Otherwise, recovery from the last committed checkpoint may attempt that individual effect again.
TCC addresses continuation recovery, not the exactly-once external I/O problem.
Related, yes, although the current prototype is narrower. Chandy-Lamport captures a consistent global state across communicating processes whereas TCC checkpoints program continuations at compiler-defined durable boundaries.
i'm building ShapelessAI, an agent that makes and publishes content, and long-running jobs dying is a real pain. The failure case I'd want demonstrated is a publish succeeding remotely, then the worker dying before committing its continuation. How does an explicit durable operation recover when the external API offers neither idempotency keys nor a way to reconcile the result?
Fun fact: The first-ever Durable Execution POC at AWS Simple Workflow used snapshots. The workflow was implemented as a fully asynchronous Java application, and a snapshot was a dump of the whole object graph using reflection. Performance was abysmal because a snapshot had to be generated after every state transition.
So we switched to replay. The biggest benefit of replay is that it lets you implement Durable Execution in any language as a library without a complex runtime. It also supports code changes while workflows are in flight (Temporal calls this patching). Making snapshots of arbitrary code state backward-compatible with code changes isn't practical.
I personally think that, in the long term, Durable Execution will use a runtime that supports both snapshotting and determinism. That way, snapshots can be taken infrequently, and replay can bring workflow code to the latest state. Similarly to a database recovering from a WAL. WASM is the most promising technology to achieve this.
I've built a framework that provides durable threads using serializable continuations in Java (with a modified JVM) and all these issues are easily solvable. Sadly it's not public :( The framework actually does have solutions for those, and all the issues raised in the article, and does support hot patching too. Plus it has a nice waitUntil() API that lets you sleep until an arbitrary combination of events including database query changes.
Performance versus log replay depends a lot on what you're doing, there are plenty of cases where snapshots are faster. Consider anything where you download a lot of data and filter it. But I argue the programming model of log replay is so terrible, and creates so many new classes of subtle bugs, that it's worth paying almost any performance price to get away from it. Especially for durable workflows correctness matters more than performance and it's much easier to achieve with continuation.
In the end it wasn't necessary (because workflows aren't expected to be super fast) but if needed I could have optimized snapshotting further with some more JVM changes. The JVM I was working with is written in Java so is easy to modify.
For hot patching there are a few tricks that help.
1. Only store live data. If a variable points to a large object graph before a checkpoint but isn't used afterwards, don't snapshot it.
2. Make it easy to switch the version of a running continuation only at known-safe checkpoints. I identify checkpoints with a (stack trace, counter) pair.
3. Mostly people want hotpatching only at specific points in their program, typically at the top of an infinite loop that's waiting for something. Design the scheduler so you can expose an API that offers "wait until something happens that I'm interested in, or I change version and then hot swap me", with a test framework that actually drives continuations through those sorts of hotswaps. If you get the API right then you (framework author) control what's live on the stack at that moment and the developer just has to think about the core state of their main root object, which they'd need to think about anyway and is where the important stuff is.
This is better than hotswap/patching with log replay, which is extremely risky - you can't change code at a given point even if you know all your workflows are beyond that point, so it's a leaky abstraction. And you just can't upgrade infinite loops at all, which makes hotswap a lot less useful to begin with.
[dead]
Good model. Anybody interested in actually training models or designing agentic systems should be doing this.
My company started around working on this problem because it's the basis for how you train programming models/reliably deploy LLMs to do specific tasks. It allowed me to build a much better mental model for LLMs because I saw how weirdly fickle/inconsistent/picky they could actually be outside of a "chat" where it feels like they have a coherent persona or consistent knowledge/capability.
Initially I thought of it as a search over prompts for capability at completing specific tasks, but now I think the speed/reliability and operations (eg can I switch models without degrading perforamnce?) benefits are even bigger benefits for most users.
A little "secret" since labs are making it harder to even use their models in this way and it's important that it be more widely understood: distribution-aware replay/re-sampling is a key technique in post-training LLMs. But it's also something that allows you to automatically identify the best model for some subset of your tasks, which can save you a lot of money.
Durable execution looks a bit like a buzzword in general. If your state is defined in the execution graph then it's just an umbrella term for a group of pre-existing algorithms and patterns. If it's undefined then what are you resuming to? The snapshot just before the crash likely leads to the undefined state again, in which case you're durably automating the crash (or even worse, uncaught incorrect behavior).
> Durable execution looks a bit like a buzzword in general. If your state is defined in the execution graph then it's just an umbrella term for a group of pre-existing algorithms and patterns.
If you first approach a tool because of buzzwords, don't be surprised that you think of the buzzwords instead of the tool.
Durable executions greatly simplify how workflows can be implemented and audited, and they literally allow eternal workflow executions that are not tied to the lifetime of an instance assigned to execute them. Durable executions do not require fancy infrastructure, only a terribly simple database.
If you understand continuations, you understand durable executions. Otherwise, you'll be stuck with the excuse that they are buzzwords.
Well, you have to either capture/eliminate/persist side effects and control the environment tightly, or it's limited in what it can do.
In a distributed or concurrent system, for full granularity, that can require specialized timing or virtualization techniques up to ensuring fully atomic snapshots and deterministic execution environments (and whether or not that properly models the SUT in real environments, or introduces bias/breaks the reproducibility in a way you care about)
Otherwise if you're only running against fixed checkpoints you have something closer to traces that maybe you could re-run or test against, in some cases, if you put in the work to set it up. In distributed systems that can be a lot of work so it's a bit vague if left unspecified. Because it's not enough to merely replay something if things can drift or don't accurately model the real system
> Well, you have to either capture/eliminate/persist side effects and control the environment tightly, or it's limited in what it can do.
You really need to read up on durable executions before commenting. Your comment reads as if you are completely oblivious to them. Their whole point is benefiting from an execution model where your workflow is comprised of idempotent pure functions whose inputs and outputs are tracked by the durable task persistence.
It's a problem I've spent a lot of time on, just not through the products marketed like temporal. I commented on the article because it's about program replay with checkpointing which is similar to what I worked with/mentioned.
I didn't see the article mention idempotency anywhere, and you didn't in the original response to the guy who said it just sounded like a buzzword; atomic snapshotting with deterministic execution is literally how you make a program continuation an idempotent function!
And the article is about solving the problem at the language runtime level so it doesn't even do that. So it would be reasonable to assume it is a buzzword if it does not have the essential property you mentioned and I was referring to. I was not even intending to disagree with you but just add what would make it less of a buzzword in this kind of case.
For the sort of architecture described in the blog post, you're persisting the program rather than a log, so you only need idempotency between any two checkpoints. For a lot of apps that's easy to achieve if write the snapshot into the same transaction the user is using for application data, and if you create/propagate idempotency tokens to other services.
You don't need specialized timing or virtualization techniques.
Well checkpointing does not make an incorrect program correct. Durability just guarantees that execution can survive infrastructure failure, not that the resumed program will not encounter the same deterministic bug again.
TCC checkpoints at explicit durable boundaries rather than at an arbitrary instruction immediately before a crash. So if the next operation repeatedly fails, ordinary retry limits or operator intervention are still required.
My proposed model is to represent the recoverable program state as a committed continuation, rather than reconstructing state by replaying the completed execution history.
Nice.
Our architecture has had somewhat of a different primitive for years that yields the same outcome.
Our application architecture is named The Feature Architecture.
A unit of work is feature. A lot of common implementation can be derived from (or compressed into) the name of a feature. They are invoked by Features.invoke(feature_name,...) and seamlessly calls same service, service to service or frontend to backend service or even backend to frontend (with client ID and userid) seamlessly.
A command feature by default does durable execution by being a consumer as well as a feature as the machinery ensures all incoming command feature messages are injected into monolog (in house built akin to kafka). So command feature errors are retried by the machinery by default.
All Dip operations are idempotent and hence can be retried maximally.
Our arcc (The architecture compiler) enforces at compile time that 1) there are zero CQRS violations of query to command invocations 2) zero violations of query to Dip.insert/update/remove (extended CQRS for persistence) 3) all Dip mutate operations are idempotent (arcc --strict), 4) zero alien Dip collection access (a feature leaf and its handlers owns exactly one collection) 5) and a whole myriad of around 10 different architecture rules that usually depend on developer discipline and conventions.
One can set a command operation to be not a durable execution by specifying bypass: true for that feature in the registry but those are the outliers.
Checkpointing and replay are not mutually exclusive in our architecture as they emerge when a command feature is either a default or one with bypass: true.
So durable executions are inherently native and first class for all commands in our Feature Architecture.
Snapshotting programs is also what https://github.com/pydantic/monty enables and aims for.
FWIW, I think we'll see a rise of AI-ready interpreters. In some sense, I like that it challenges traditional microservice architectures as an aside.
Yes, Monty is highly relevant. Its ability to serialize and resume a paused interpreter has a clear relationship to this work. I would say the main difference is scope; Monty is a secure Python interpreter for AI-written code, while TCC is exploring durable execution semantics around effects, waits, child executions, and persisted continuations, eventually across language frontends. Thanks for the insight, I'll definitely add Monty to the related-work discussion.
I’ve been building a durable execution framework with roughly the same mechanism. It’s a pretty logical rough edge to try shave off from Temporal-like systems.
Congratulations.
I am curious about what prompted you to build a durable execution framework.
We didn't know we were building one until later as that is where our application development trajectory naturally lead us.
Good question. I operate a pretty big Temporal deployment at work (~3.45 billion actions per month), and also have the dubious distinction of being one of the only community SDK (Haskell) implementors for Temporal. So I have a pretty good sense of rough edges and my own areas of dissatisfaction, and also what I'd want out of a next generation system.
So, hello frenemies, I suppose :)
Which rough edges pushed you toward the same mechanism - history growth, replay latency, determinism/versioning, worker ops, or the programming model? Happy to compare notes.
Definitely friends, not frenemies. I have nothing but genuine respect for builders out there solving hard infrastructure problems.
We are primarily focused on our Feature Architecture. Durable execution is only for command features, requires tight coupling with our architecture, and is not a standalone dedicated tool like Temporal.
Open to further discussion on both of our work's trajectories rather than digress here.
How did you know you need one?
Good question.
Building for extreme scale constrained the possible architectural space. For a high level overview of the application architecture, refer to the seperate comment here in this thread.
It started initially by realizing that our app, Slyp, requires extremely distributed invoice and coupon processing as we started rolling out. So the architecture must handle such scale without issues.
This lead to persisting every incoming request (for commands) from the frontend and dismissing them only to be informed later of continuing with the status. This avoids processing requests when the system is overloaded with high traffic. So naturally the minimum is a log style ordered persistence. Initially kafka. Later our own Monolog built in rust which is substantially faster for our workloads and zero jvm and low memory and can run on servers and mobile alike.
Naturally the right point to consume this was into the command ingestion point in the Feature Architecture as noted in the other comment. This expanded the capability to all feature invocations including inter and intra service feature invocations.
This then progressively evolved into a much capable engine at which point we realized, this is a durable execution engine for command features but as a minor part of the whole Feature Architecture itself.
Was this written by AI? I even read the "technical paper" and still don't understand what it's supposed to do. Nowhere does it explain how this "checkpointing" and "resuming a continuation" actually works in practice.
Fair point. The paper explains the execution model and evaluation, but I should have included a concrete source-to-continuation transformation. I kept the implementation discussion too abstract while the design was still evolving.
Essentially, the compiler lowers the program into an explicit state machine, and a committed checkpoint contains the next state identifier along with live values and any relevant control states needed to resume execution. For recovery, a fresh runtime hydrates that frame and resumes directly from the saved state rather than replaying the completed history.
Checkpointing the live continuation shifts recovery cost toward retained state rather than lifetime event count, which seems especially useful for agents with many completed tool calls.
Kind of reminds me of how redis forks itself to have a snapshot to persist as a backup.
I always found that to be really elegant.
Yup, essentially capture state now so execution can continue from it later. The difference is that this checkpoint represents compiler-known program state (live values and control position) rather than an entire process or address space. That lets it omit dead state and the completed execution prefix.
Another thing it brings to mind is Record Replay (RR) debugger that was originally made for Firefox.
BTW, for some reason a lot of your comments are dead. I'm bring them back.
dropping history replay would remove so much operational pain from durable workflows. how do you reconstruct in-flight state after a crash, snapshotting or something event sourced
Closer to compiler-generated continuation checkpointing than event sourcing.
In the current prototype, the compiler lowers the program into an explicit state machine, and then at each durable boundary - waits, external effects, child executions, etc - it commits the next program position along with live locals, control state, and any values needed to continue execution.
When the program crashes or is killed while executing, a fresh runtime loads the checkpoint, reconstructs the frame, and dispatches directly to the saved position - no history replay.
External effects still have the usual ambiguity window: an operation may succeed externally, but the process may crash before its result and the updated checkpoint are durably committed. To solve this, TCC gives each effect a stable identity, but non-idempotent operations still require provider-supported idempotency or reconciliation. Otherwise, recovery from the last committed checkpoint may attempt that individual effect again.
TCC addresses continuation recovery, not the exactly-once external I/O problem.
Lamport & Chandy, right?
Related, yes, although the current prototype is narrower. Chandy-Lamport captures a consistent global state across communicating processes whereas TCC checkpoints program continuations at compiler-defined durable boundaries.
[flagged]
i'm building ShapelessAI, an agent that makes and publishes content, and long-running jobs dying is a real pain. The failure case I'd want demonstrated is a publish succeeding remotely, then the worker dying before committing its continuation. How does an explicit durable operation recover when the external API offers neither idempotency keys nor a way to reconcile the result?
[dead]