
Durable Execution for AI Agents: Architecture and Failure Handling
An AI agent that works in a demo often breaks in production for a boring reason: the process running it dies halfway through. A deploy restarts the container, a model API times out, a tool call hangs, or a person takes two days to approve a refund. If the agent's progress lives only in memory, you lose the run, or worse, you retry it and repeat side effects. Durable execution for AI agents is the engineering answer to that problem: every step of the agent loop is recorded so a run can resume exactly where it stopped. This article covers how to design it: the data model, the step loop, side effects, human approvals, and the trade-offs between building it yourself, using a framework checkpointer, or adopting a workflow engine.
If you are still deciding what agents should do in your company and how to roll them out, start with our 7-step guide to AI agents for business. This piece assumes that decision is made and focuses on the system underneath.
Why agent runs fail differently from normal requests
A typical web request is short, mostly deterministic and easy to retry. An agent run is none of those:
- It is long. A run can involve several model calls and tool calls, and may pause for minutes or days waiting for a human.
- It is non-deterministic. Calling the model again with the same input can produce a different plan. You cannot simply rerun from the start and expect the same path.
- It has side effects. Tools send emails, create invoices, update a CRM record or issue refunds. Repeating them is a business problem, not just a technical one.
- It depends on flaky dependencies. Model providers rate-limit and time out, and internal APIs go down.
Temporal's own framing is useful here: in announcing its OpenAI Agents SDK integration, it states plainly that "AI agents are distributed systems". Treat them like one: assume every step can fail, and design so that failure costs a retry instead of a lost run.
What durable execution for AI agents actually means
The core idea is to split the agent into two kinds of code:
- Orchestration: the loop that decides what happens next (call the model, run a tool, wait for approval, finish). This code must be replayable.
- Steps: the I/O-heavy work, such as model requests, tool calls and external APIs. Each step's result is persisted once it completes.
When the process crashes, a new worker loads the run's history, replays the orchestration using the recorded results instead of calling the model again, and continues from the first step that never finished. The Pydantic AI documentation describes the same split for its Temporal integration: model requests, tool calls that may require I/O, and MCP server communication are offloaded to Temporal activities, while the agent's coordinating logic runs in the workflow.
That rule, "never redo a step whose result is already recorded", is what keeps a non-deterministic model from taking a different path after a restart.
The data model if you build it yourself
You do not need a workflow engine to get most of the benefit. A relational database and a strict step loop go a long way. A minimal schema:
CREATE TABLE agent_runs (
id UUID PRIMARY KEY,
agent_name TEXT NOT NULL,
status TEXT NOT NULL, -- running | waiting_approval | completed | failed | cancelled
input JSONB NOT NULL,
lease_owner TEXT, -- worker currently executing the run
lease_expires TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE agent_steps (
run_id UUID NOT NULL REFERENCES agent_runs(id),
step_index INT NOT NULL,
kind TEXT NOT NULL, -- model_call | tool_call | approval | final
request JSONB NOT NULL, -- prompt or tool name + validated arguments
result JSONB, -- null until the step completes
status TEXT NOT NULL, -- pending | completed | failed
attempts INT NOT NULL DEFAULT 0,
idempotency_key TEXT NOT NULL UNIQUE,
PRIMARY KEY (run_id, step_index)
);
CREATE TABLE agent_approvals (
run_id UUID NOT NULL,
step_index INT NOT NULL,
decision TEXT, -- approved | rejected | edited
edited_args JSONB,
decided_by TEXT,
decided_at TIMESTAMPTZ,
PRIMARY KEY (run_id, step_index)
);
Three design choices matter more than the exact columns:
- Steps are append-only and ordered. The pair
(run_id, step_index)is the step's identity. Replaying a run means walking these rows in order. - The idempotency key is derived, not random. Use something like
run_id:step_index. If a worker crashes after calling a tool but before saving the result, the retry sends the same key, and a well-behaved downstream API can deduplicate it. - A lease prevents two workers from running the same run. A worker claims a run by setting
lease_ownerandlease_expiresin a conditional update, and renews the lease while it works. If it dies, the lease expires and another worker picks the run up.
The step loop
The orchestration loop is where correctness lives. In pseudo-code:
def execute_run(run_id, worker_id):
if not claim_lease(run_id, worker_id):
return # another worker owns it
history = load_completed_steps(run_id) # ordered list
messages = rebuild_messages(history)
step_index = len(history)
while True:
# 1. Ask the model what to do next (a recorded step)
decision = run_step(run_id, step_index, "model_call",
request={"messages": messages},
fn=lambda: call_model(messages, tools=ALLOWED_TOOLS))
step_index += 1
if decision.type == "final":
mark_run(run_id, "completed", output=decision.content)
return
# 2. Validate tool arguments before anything runs
tool = TOOLS[decision.tool_name]
args = tool.schema.validate(decision.arguments)
# 3. Pause for a human if the tool is sensitive
if tool.requires_approval:
approval = get_approval(run_id, step_index)
if approval is None:
request_approval(run_id, step_index, tool.name, args)
mark_run(run_id, "waiting_approval")
release_lease(run_id)
return # the process exits; nothing is held in memory
if approval.decision == "rejected":
messages.append(tool_rejected_message(tool.name))
step_index += 1
continue
args = approval.edited_args or args
# 4. Execute the tool exactly once per step (a recorded step)
result = run_step(run_id, step_index, "tool_call",
request={"tool": tool.name, "args": args},
fn=lambda: tool.execute(args,
idempotency_key=f"{run_id}:{step_index}"))
messages.append(tool_result_message(tool.name, result))
step_index += 1
def run_step(run_id, step_index, kind, request, fn):
existing = get_step(run_id, step_index)
if existing and existing.status == "completed":
return existing.result # replay: never call the model or tool again
upsert_pending_step(run_id, step_index, kind, request)
result = with_retries(fn, max_attempts=3, backoff="exponential")
complete_step(run_id, step_index, result)
return result
Notice what the loop never does: it never holds state that only exists in memory, and it never re-executes a completed step. Waiting for a human is not a sleeping thread; it is a status in the database. When the approval arrives, a webhook or queue message triggers execute_run again and the loop replays its way back to the same point.
Side effects: the part that actually hurts
Durable execution guarantees that your orchestration resumes correctly. It cannot guarantee that an external system did not receive a request whose response you never saved. That gap is where duplicate emails and double refunds come from. Close it with three patterns:
- Idempotency keys on every write tool. Pass the derived key to downstream APIs that support it. For your own services, store processed keys and return the original result when a key repeats.
- Transactional outbox for internal effects. When a tool writes to your own database and publishes an event, write the record and the outgoing message in the same transaction, then let a separate process deliver the message.
- Compensation for things you cannot make idempotent. If a tool has no deduplication, design a reverse action (cancel the order, void the invoice) and record which steps need it if the run is abandoned.
Classify every tool as read-only, idempotent write, or non-idempotent write before you give it to an agent. Read-only tools can retry freely. Non-idempotent writes should usually require approval or be wrapped in a service you control.
Human approvals as durable waits
Approvals are the clearest reason persistence is mandatory, not optional. The LangChain documentation for its human-in-the-loop middleware states that you must configure a checkpointer to persist the graph state across interrupts, and recommends a persistent checkpointer such as AsyncPostgresSaver or MongoDBSaver in production, keeping InMemorySaver for testing.
Whatever stack you use, design the approval record with care:
- Show the reviewer the exact validated arguments, not the model's prose about them.
- Allow three outcomes: approve, reject with a reason the agent can read, or edit the arguments.
- Record who decided and when. This is your audit trail.
- Set an expiry. A run that waited too long should fail cleanly instead of executing on stale data.
Build, framework or workflow engine: the trade-offs
There are three realistic approaches, and the right one depends on how long your runs are and how much infrastructure you want to own.
1. Do it yourself on Postgres
Good for: a few agents, short-to-medium runs, teams that already operate Postgres and a queue. Pros: full control, no new infrastructure, the data model fits your domain. Cons: you own leases, retries, timers and replay logic, and subtle bugs in those are hard to find.
2. Framework checkpointers (for example, LangGraph)
Good for: teams already building on that framework. The LangGraph docs explain that checkpointers save state so a person can inspect, interrupt, and approve graph steps and the graph can resume afterwards. They also expose durability modes; in the "exit" mode, changes are persisted only when graph execution exits, whether successfully, with an error or at an interrupt. That is cheaper but means a crash mid-run can lose the progress since the last exit, so choose the mode per workload. Cons: durability is scoped to the framework's graph; side effects inside tools are still your responsibility.
3. A durable execution engine (for example, Temporal)
Good for: long-running, high-value or multi-agent workflows. Temporal's tutorial on durable agents notes that workflows preserve context through system interruptions and that the duration of the Workflow Execution is irrelevant. Temporal offers integrations with the OpenAI Agents SDK (announced in public preview) and with the AI SDK by Vercel. Cons: another system to run or pay for, and the workflow code must follow the engine's determinism rules, which takes some getting used to.
A pragmatic path: start with approach 1 or 2 for a single, well-scoped agent, and move to an engine when you have long waits, many concurrent runs or cross-service workflows.
Security and cost controls that belong in the architecture
Durability makes agents resilient; it also makes them persistent, which raises the stakes of a bad action. Build these into the loop, not into the prompt:
- Allow-list tools per agent and per run. The model can only call what
ALLOWED_TOOLScontains. - Validate arguments with a schema before execution and before approval. Reject, do not coerce.
- Use scoped credentials. Each tool runs with the narrowest permissions it needs, never a shared admin token.
- Set budgets. Cap the number of steps, model tokens and wall-clock time per run, and fail the run when it exceeds them.
- Keep large payloads out of the step table. Store documents and long tool outputs in object storage and save a reference, so replay stays fast and checkpoints stay small.
Common mistakes
- Keeping conversation state in process memory "for now".
- Retrying the whole run from the start after a failure, which re-calls the model and can pick a different path.
- Generating a random idempotency key on each attempt, which defeats the purpose.
- Treating approval as a blocking call in a web request.
- Letting tools decide their own permissions instead of enforcing them in the orchestrator.
- Logging only the final answer, so you cannot explain how a run reached it.
Implementation checklist
- Every run has a persistent ID and status.
- Every model call and tool call is a recorded, ordered step.
- Completed steps are never re-executed on replay.
- Idempotency keys are derived from run and step, and passed downstream.
- Each tool is classified as read-only, idempotent write or non-idempotent write.
- Sensitive tools require approval, with expiry and an audit record.
- A lease or engine guarantees one active worker per run.
- Retries use backoff and a maximum attempt count.
- Step, token and time budgets are enforced in code.
- Large payloads live outside the checkpoint store.
If you need this kind of backbone built into your own product, with your data model and your systems, that is the work our team does in custom software development.
Frequently asked questions
Do I need Temporal to make an AI agent durable?
No. A database with an ordered step table, derived idempotency keys and a lease per run covers many cases. An engine becomes worth it when runs are long, numerous or span several services.
Is a framework checkpointer the same as durable execution?
It covers part of it: saving and resuming agent state, which is what human approvals need. It does not make side effects inside your tools safe; idempotency and compensation are still your job.
How do I stop a resumed agent from repeating an action?
Never re-execute a completed step, and send a deterministic idempotency key with every write so the downstream system can ignore duplicates when the crash happened between the call and the save.
Comments
No comments yet. Be the first to share what you think.


