Agents That Crash and Restart From Zero Are Not Production Agents

A 429 from a downstream API mid-workflow should cost you a retry. For most teams that shipped multi-step agents in 2024 and 2025, it cost the whole task. The orchestrator had no notion of where it was in the workflow, so the only recovery path was starting over. At best that means a duplicate side effect: a refund issued twice, a confirmation email sent twice. At worst the user watches an agent that was three steps from done forget everything and start asking the same questions again.

State was never in the design doc. It was an implementation detail nobody budgeted for, right up until it became the incident nobody budgeted for either. Teams are now retrofitting checkpoint layers under production load instead of designing them in from day one. That retrofit is more expensive, slower, and riskier than doing it up front, and it's happening across the industry right now.

Checkpoint-based recovery is baseline, not advanced

AWS's newly published Agentic AI Lens (AGENTREL03, "Agent memory and state management," part of the six-pillar Well-Architected framework AWS introduced at re:Invent 2025) frames state handling as a maturity progression rather than a one-time architecture decision. The best-practice set includes information classification, fault-tolerant memory stores, checkpoint-based recovery, and graceful degradation, organized as tiers a team climbs, not a checkbox you tick once. The headline point worth internalizing: checkpoint-based recovery sits inside that baseline set, not in some "advanced/mature org only" tier reserved for teams with unlimited platform budget. If your longest-running agent workflow doesn't have it, you are, by AWS's own framing, below the maturity bar for production agentic systems.

Here's the sharpest failure mode once you decide to add checkpoints: checkpoint replay without idempotent steps doesn't produce safe resumption. It produces duplicate side effects. If step 4 of 7 calls a payment API and the workflow crashes on step 5, a naive "replay from last checkpoint" resumes at step 4 and calls the payment API again. Idempotency has to be designed alongside checkpoints, not bolted on after the first duplicate-charge incident teaches you the lesson.

The concrete version of "session scope vs. persistent scope" shows up in Amazon's own architecture writeup on durable LangGraph agents. LangGraph's default InMemorySaver keeps checkpoint state in the process. That's session scope: fast, zero setup, and gone the moment the process restarts. Swap in a DynamoDBSaver and the checkpoint survives the process.

With in-memory storage, you lose progress. With DynamoDBSaver, the workflow can query the last successful checkpoint and resume from there.

— Hannigan — Build durable AI agents with LangGraph and DynamoDB (AWS, 2026)

That's the entire difference between a demo and a production system in one sentence. Most teams ship the demo's checkpoint strategy to production because it's the framework default, and nobody flips the switch until a restart under load turns into a support ticket.

Flat memory makes recovery expensive; a state tree makes it structural

Checkpointing tells you where the workflow crashed. It doesn't tell you which prior turns were actually on the path that succeeded versus the ones the agent abandoned mid-attempt. That distinction is where most teams' memory design quietly fails them, and it's a different problem from "do we have a checkpoint at all."

The MAGE paper (Chen et al., June 2026) names the failure mode precisely: standard RAG-style semantic memory retrieves past turns by similarity, and that approach "fragments decision trajectories and mixes valid and erroneous traces." A flat memory store can't tell a completed sub-task from an abandoned one, because similarity search doesn't encode execution structure. It just encodes "these two things sound related."

MAGE's fix is a hierarchical state tree that mirrors the agent's actual execution: parent task, sub-tasks, retries, all structurally related rather than semantically clustered. Retrieval respects the dependency graph the agent actually walked, instead of pattern-matching on vocabulary. The numbers behind this are the headline of the paper, and they're worth stating precisely rather than rounding up:

MAGE improves the average task success rate by 7.8-20.4 pp over baselines, while reducing token consumption by 55.1%.

Say "up to 20 percentage points," not "an average of 20." The range runs 7.8 to 20.4pp; 20.4 is the ceiling, not the midpoint. The 55.1% token reduction is the average figure and can be stated as-is.

Here's the part engineering leaders skip past too fast: this isn't only a retrieval-quality problem. It's a recovery-cost problem. When your workflow resumes from a checkpoint, the agent has to reconstruct which prior turns were on the successful branch before it can decide what to do next.

A tree encodes that structurally. The parent-child relationship is the data, not an inference. A flat store forces the model to re-derive it from scratch on every resume.

That re-derivation burns tokens on every single recovery, and it reintroduces whatever errors lived in the abandoned branches it can't distinguish from the live one. You pay this tax every time a workflow restarts, which, if you skipped Section 1, is often.

Mapping the maturity tiers to what you build this quarter

Three concrete forks, in the order teams actually hit them:

Walk one workflow through all three and the maturity model stops being abstract. Take the payment-API example from Section 1: step 4 of 7 charges a customer, step 5 crashes on a transient 429 from the shipping carrier's API.

At session scope with no idempotency, this is a full restart. The InMemorySaver died with the process, so the workflow has no memory of reaching step 4 at all. It starts at step 1, re-collects the order, and re-calls the payment API. The customer is charged twice. This is the failure mode from the opening of this post, and it's still the default most teams ship.

Move to persistent scope, still no idempotency, and you've fixed half the problem. The DynamoDBSaver checkpoint says "step 4 completed." The workflow resumes at step 5 instead of step 1, which is real progress: no re-collecting the order, no re-running steps 1 through 3. But if the crash happened during step 4, mid-payment-call, before the checkpoint write landed, the replay logic can't tell "step 4 completed" from "step 4 was in flight." It resumes at step 4 and calls the payment API again anyway. Persistence alone moves the bug later in the workflow. It doesn't remove it.

Add idempotency keys, and step 4 checks "have I already charged this order with this key?" before calling the payment API, regardless of whether the checkpoint says the step completed or was merely attempted. Now the replay is safe whether the crash happened before, during, or after the checkpoint write. This is the combination AWS names directly as its own best practice, AGENTREL03-BP03, "implement comprehensive state management and checkpoint-based recovery": persistence and idempotency are one requirement, not two independent nice-to-haves you can ship separately and call done.

Layer in the state tree, and the fourth failure mode disappears too: the one where the workflow resumes correctly at step 5, but the agent's memory of why it's at step 5, which sub-task this belongs to, what the abandoned retry from step 3 looked like, is a flat log the model has to re-parse instead of a structure it can walk. The tree is what makes the resumed agent's next decision as good as the same decision made mid-session, instead of a slightly-confused guess made from a pile of undifferentiated history.

None of this is novel engineering. It's engineering most agent teams skipped because the framework tutorial didn't show it, and because each piece looks optional in isolation. Stacked together, on one real workflow, they're the difference between "the customer got charged once and the package still shipped" and the incident review nobody wants to write.

// Minimum contract for a resumable checkpoint. The idempotencyKey
// is what turns "resume" into "resume safely" instead of "replay
// side effects." The tree lives in the memory store this checkpoint
// points into, not in the checkpoint payload itself.
type Checkpoint = {
  workflowId: string
  stepIndex: number          // last successfully completed step
  scope: "session" | "persistent"
  idempotencyKey: string     // checked before re-executing any side effect
  payloadRef?: string        // S3 pointer if payload > 350KB
  ttl: number                // epoch seconds; expire abandoned sessions
}

This is the same underlying pattern we flagged in /blog/your-agent-passes-the-benchmark-it-will-fail-in-production under "state-evolution faults," where an agent's internal memory drifts from reality across turns and nothing in the architecture catches it. Crash recovery is the acute version of that same chronic problem: the same undesigned state that quietly corrupts a long session is what makes a clean restart impossible when the process actually dies.

What to check Monday morning

Pick your longest-running agent workflow. Ask one honest question: if step 4 of 7 throws a transient error right now, what happens? If the answer is "it restarts from step 1," you have a session-scope memory system dressed up as a production system, and the AWS maturity model would put you below baseline, not on some advanced tier you haven't gotten to yet.

The fix isn't a bigger model. It's three concrete, boring engineering decisions: a persistent checkpoint store instead of the framework default, idempotency keys on every side-effecting step, and a state representation, a tree, not a flat log, that survives the resume instead of forcing the model to re-derive it every time. None of the three requires a research breakthrough. All three require someone to have put it in the plan before the first production incident forced the retrofit.

References

• AWS Well-Architected Agentic AI Lens — AGENTREL03: Agent memory and state management
• Chen et al. — MAGE: Beyond Semantic Organization (2026)
• Lu et al. — FluxMem: Choosing How to Remember (2026)
• Hannigan — Build durable AI agents with LangGraph and DynamoDB (AWS, 2026)