June 2, 2026 ·

Agent harness design: the persistence fabric

June 2, 2026

This is a companion to Agent harness design: trade-off analysis, which frames the agent platform as a small set of separable primitives connected by contracts. That overview names four primitives the ecosystem commonly agrees on, sandbox, persistence, identity, and the harness itself. This post goes deep on the second of them: the persistence fabric.

The persistence fabric is the durable-state primitive. Its contract is small and uniform across everything it stores: append, read, snapshot, fork. The things it stores are not: the session log (append-only intent), the workspace (mutable sandbox state), long-term memory (curated and retrievable), and artifacts (user-facing deliverables). One contract, four very different state shapes, each with its own durability guarantee and its own failure modes. The argument of this post is that those guarantees are not interchangeable, that the gaps between them are where production data loss and cross-tenant leaks actually happen, and that getting the contract right is mostly about being honest about what each store does not guarantee.

The contract, and the seam with identity

The durable-state contract has four verbs and four state shapes, but it has one invariant that runs underneath all of them: every resource the fabric stores is identity-owned. Each session log entry, each workspace, each memory record, each artifact blob, and each audit entry carries an owner, a tenant, and an ACL sourced from the identity fabric. The durability contract captures what happened and where it lives. The identity contract captures for whom, against whose tenancy, under what scopes. They are different questions answered by different primitives, and two of the fabric's stores, memory and audit, are genuinely co-owned: they are persistence in mechanism and identity in policy.

This post does not re-derive identity. It states the seam where it matters (per-tenant memory partitioning, right-to-erasure cascades, audit attribution) and points back to the overview for the tenancy and ownership model. What follows is the persistence side: the mechanisms, the trade-offs, and the failure modes of each store.

Durability models: what gets recorded, not just where

A harness has to survive any of: a harness crash, a replica reschedule, a sandbox pod loss, or a user resuming a session days later. There are three durability models, and the meaningful difference between them is what they record, not where they put it.

Append-only event log (intent)

Every user message, model response, tool call, and tool result is written to a durable store. The session log is the sole recovery artifact: any replica can serve any session by reading the log. This buys clean replay, clean audit, and transport-agnostic recovery.

The failure mode is the gap between a side effect and its commit. The log records intent ("call git commit") and then, separately, result ("commit succeeded"). If the harness crashes in the window between those two writes, recovery reads the last durable state, which is pre-result, and the agent re-issues the operation. For an idempotent operation that is harmless. For a non-idempotent one, a second git commit, a duplicate ticket, a repeated payment, the session store and the external world diverge silently, and nothing in the log can tell you it happened.

sequenceDiagram
    participant LLM
    participant R as Runner
    participant SB as Sandbox
    participant SS as Session store

    R->>LLM: prompt
    LLM-->>R: tool_use: sandbox_exec("git commit -m 'feat'")
    R->>SB: gRPC Exec("git commit ...")
    SB-->>R: OK (commit created)
    Note over R: ✕ Runner crashes here
    Note over SS: Event NOT persisted

    Note over R,SS: Recovery
    R->>SS: GetSession
    SS-->>R: last known state (pre-commit)
    R->>LLM: replay from last checkpoint
    LLM-->>R: tool_use: sandbox_exec("git commit -m 'feat'")
    R->>SB: gRPC Exec("git commit ...")
    Note over SB: ✕ Duplicate commit or error:<br/>nothing to commit, working tree clean

The deeper point is that an intent log records what was requested, and recovery cannot distinguish "requested but never ran" from "ran but never recorded." Both look identical from the log: a tool call with no result. Retries and replay are common enough in practice (reported agent retry rates of 15 to 30 percent) that these operations have to be designed for, not treated as edge cases [1].

Durable step log (intent plus a commit boundary)

The step log closes the gap by adding a third record between intent and result: a durable step boundary written before the effect executes. Frameworks like Restate [2], Temporal [3], and Inngest [4] journal each step, run the effect, then record its outcome; replay skips any step whose outcome is already durable. The model's value is that it formalizes the distinction the intent log blurs, "we asked to do X" and "X completed" are now two separate, individually-durable facts, so replay can ask "did this step already complete?" instead of assuming it did not.

The mechanism that makes this work at the external boundary is an idempotency key threaded through the step. The journal records the step's intent and its key before the effect fires; on replay the harness performs check-then-act against that key (or against the external system's own dedup, git's ref state, a payment processor's idempotency header) rather than blindly re-issuing. The step log does not make external effects idempotent on its own. It gives you the hook to make them so. Temporal's own framing is that probabilistic LLM behavior makes naive retry logic insufficient for agent workflows, which is exactly the case durable-execution journaling is built for [5]. By 2025 this had crossed into the mainstream: AWS Durable Functions, Cloudflare Workflows, and Vercel's Workflow DevKit all shipped with AI-agent use cases as primary framing [4]. The pattern is no longer framework-specific.

Filesystem-as-state

The third model takes an extreme position: the working context (open files, partial plans, execution checkpoints) is ephemeral and disappears the moment the context window resets or a process is interrupted, so the filesystem should be the sole authoritative record of task state [6]. The event log demotes to a secondary log of intent; the filesystem holds the truth. In practice this pushes the design toward an opinionated convention (progress.md, plan.md, journal.md) that the agent is trained or instructed to maintain.

This model is appealing because it sidesteps the intent-versus-result gap for anything the filesystem can represent: the file either has the new content or it does not, and that is observable on recovery. But it relocates the gap rather than removing it. Effects that the filesystem cannot record, a git push to a remote, an external API call with side effects, an MCP server-side cursor advance, are exactly the non-idempotent operations the intent log struggled with, and they are still outside the durable record. Filesystem-as-state moves authority to the filesystem; it does not extend the filesystem's reach over the rest of the world.

The reconciliation gap

Whichever model is chosen, one question does not answer itself: if the sandbox holds mutable state (files, packages, processes) outside the durable log, how does the harness reconcile the two on recovery? For idempotent operations (overwrite a file with known content) it does not matter, the result is the same whether the operation ran once or twice. For non-idempotent ones (append to a file, increment a counter, create a commit, call a side-effectful API), the harness needs either a step-log boundary or an idempotency key threaded through the tool. This is the single most important property to nail down before the first crash, because it is invisible until exactly the wrong moment, and by then the divergence has already shipped to the outside world.

Workspace persistence: partial durability, stated honestly

The workspace, the sandbox's mutable filesystem, is state that outlives individual tool calls but is not on the durable log. Persisting it across pod lifecycle events is an independent axis with its own trade-offs.

PersistentVolumeClaim (PVC)

The common Kubernetes pattern mounts a PVC at /workspace; when the pod dies, a replacement mounts the same PVC. The dominant failure is zone scoping. A ReadWriteOnce PVC is backed by a zonal block device, and a zonal block device is physically attachable only to a node in its own availability zone. So when the pod reschedules across zones (node failure, capacity rebalancing, spot eviction), the volume cannot follow the pod, and recovery falls back to a full re-bootstrap. This is not a configuration mistake; it is the storage topology. Rack2Cloud documents it as a routine day-2 failure: a pod restarts on a different node, the new node cannot mount the PVC, and the pod sits in ContainerCreating [7]. Kubernetes issue #121436 shows PVC binding annotations persisting after a failed schedule and blocking rescheduling to viable nodes [8]. RWO also creates a teardown race during crash recovery: the old pod must release the volume before the new one can mount it, so an old pod terminating blocks the replacement until the grace period elapses, adding seconds to minutes of unpredictable latency.

When the pod is deliberately destroyed (teardown policy, session timeout, scale-down), the PVC is often deleted with it, and any uncommitted work is gone permanently. Gitpod issue #9544 is the canonical case: a workspace timeout fired before the final sync completed and a user lost roughly two hours of work [9]. A newer variant is Karpenter issue #2777 (2026): Karpenter injects the PVC's zone into pod NodeAffinity, which can leave pods indefinitely Pending after node deletion because the only node that could satisfy the affinity has been removed [10].

Snapshot and checkpoint

The alternative is to checkpoint the complete state, workspace, conversation, and environment, on a cadence and restore from the checkpoint on crash. Replit's snapshot engine implements this as a copy-on-write filesystem plus versioned database state, explicitly capturing the AI conversation context alongside the filesystem, and reports recovering from OOM crashes that occurred roughly once an hour without data loss [11] [12]. The cost is a direct trade-off between snapshot frequency and storage overhead, plus the operational weight of a checkpoint pipeline. The frequency knob is load-bearing: it sets the maximum amount of work a crash can erase.

Reproducible bootstrap as recovery

A third option treats the sandbox as fully disposable and rebuilds state from a deterministic bootstrap spec (pinned commit SHAs, locked package versions, side-effect-free setup). This works only if the bootstrap genuinely is deterministic. A floating git clone main, an npm install without a lockfile, or a setup script that mutates remote state turns "resume" into an operation that drifts a little every time, so the recovered workspace is not the one that was lost.

The honest statement

Workspace persistence is partial durability, not full recovery. Any design that claims "sessions survive indefinitely" owes an explicit account of which failure modes it covers (pod restart in the same zone) and which it does not (zone failure, teardown before sync, preemption of a hibernated node). The honest version of the claim names the gaps; the dishonest version discovers them in an incident.

Branching and fork: the part of persistence that is a tree

A linear append-only log treats the session as a list. Production harnesses have moved past that: Claude Code [13], Cursor [14], Replit Agent [15], LangGraph [16], and OpenAI's Codex CLI [17] all ship some form of rewind, checkpoint, or fork, turning the session from a list into a tree (or at least a log with a movable head). That changes the data model, and it introduces trade-offs that are independent of plain durability.

The options run from linear-only (immutable trajectory; to back out, start over) through rewind/truncate (jump back to a checkpoint and discard the tail, with the workspace rolled back to match, as in Claude Code's /rewind and Cursor's per-edit checkpoints) to a forkable tree (multiple live branches from any checkpoint, all preserved; LangGraph's time-travel model treats checkpoints as a DAG with explicit branch IDs [16], and Claude Code's /fork spawns a child session from a shared history point [13]).

The workspace-branching problem

graph LR
    classDef easy fill:#d5f5e3,stroke:#27ae60
    classDef hard fill:#f9d6d6,stroke:#c0392b

    CK[Checkpoint]
    CK --> B1[Branch A: conversation]
    CK --> B2[Branch B: conversation]
    CK --> W[Workspace / sandbox state]
    W --> C{Shared or cloned?}
    C -->|shared| RACE[Branches interfere]
    C -->|cloned| COST[Clone cost per fork]

    class B1,B2 easy
    class RACE,COST hard

Branching conversation state is cheap, because it is just a tree of event-log IDs and forking a tree is a pointer operation. Branching workspace and sandbox state is not, because the workspace is a real filesystem and two branches cannot both own it. This is where the storage substrate (below) becomes load-bearing: copy-on-write overlays and content-addressed snapshots make a fork cheap because only the divergent blocks are duplicated, while a PVC has no native cheap-clone primitive at all. The choice collapses to a dilemma: forks that share the workspace race on it (branch A's writes are visible to branch B), and forks that clone the workspace pay the clone cost, which is bounded by the substrate, not by the harness.

Failure modes, and why merge-back is hard

  • External state divergence. A branch can fork its conversation and its workspace, but it cannot fork the external systems it touches: git remotes, issue trackers, MCP server-side cursors. An agent on branch A may push a commit that branch B then tries to push and conflicts with. The branchable state and the unbranchable state drift apart.
  • Merge-back is genuinely unsolved, and the asymmetry is structural. File merge is tractable because files have a common ancestor: the checkpoint they forked from gives a three-way merge base, so a standard ancestor-versus-A-versus-B merge applies. Dialogue has no such base. There is no meaningful three-way merge of two divergent conversations, because the "content" is a reasoning trajectory, not a set of editable lines. This is why the agent, not the harness, has to perform the file merge: the harness can supply the three versions, but only something that understands the task can resolve the semantic conflict.
  • Orphan branches as cost leaks. Every abandoned branch pins workspace, sandbox, and session-log resources until a retention policy reclaims them. Without a branch TTL, orphans accumulate silently, and because each one is small the leak is easy to ignore until the aggregate is not.
  • Checkpoint granularity. Too fine (one checkpoint per edit) produces snapshot storms; too coarse (one per user turn) loses intra-turn recovery points. The right granularity is a function of how expensive a snapshot is on the chosen substrate, which ties this axis directly to the next one.

Rewind alone suffices for interactive coding. Forkable trees pay off when users explore alternatives in parallel, when an evaluation pipeline replays a session with a changed prompt, or when the harness is itself an experimentation platform. Linear-only is defensible for strictly transactional workloads.

Artifact surface: deliverables outlive workspaces

An artifact is a generated file, image, build output, or structured result that is a user-facing deliverable, distinct from the workspace, which is internal state. Replit markets CSVs, PDFs, slide decks, and Markdown as first-class deliverables alongside the built app [18]; OpenHands captures produced files through its actions/observations event stream [19]; OpenAI's Responses API returns images, files, and structured outputs as attachments distinct from message content [20]; E2B sandboxes expose a downloads API for files produced during execution [21].

The design choice is how the artifact is surfaced: in-workspace-only (everything under /workspace; the client must guess which paths are deliverables), artifact events on the session log (the agent emits an explicit artifact event with blob ID, MIME type, filename, and description, backed by a blob store with its own retention policy), or a typed artifact contract (a taxonomy of image, file, table, chart, with per-type client renderers). The middle option is the usual production answer, and it carries the key insight: artifact retention almost always has to outlast workspace retention, because the workspace is torn down when the session ends but the PDF the agent produced is still the user's deliverable. An artifact pipeline that piggy-backs on workspace storage inherits the workspace's TTL, and that is a bug.

Lineage is the difference between an artifact and an opaque blob

graph LR
    classDef art fill:#d5f5e3,stroke:#27ae60
    classDef meta fill:#fdebd0,stroke:#e67e22

    ART[Artifact: report.pdf]
    ART --> CK[Checkpoint that produced it]
    ART --> SESS[Originating session]
    ART --> TOOL[Tool call: pandoc_convert]
    ART --> IN[Input: draft.md]

    class ART art
    class CK,SESS,TOOL,IN meta

A serious artifact system records which checkpoint produced the artifact, from which inputs, via which tool call. That lineage is simultaneously the reproducibility story (you can regenerate it) and the audit trail (you can attribute and, when required, selectively delete it). Without lineage, an artifact is an opaque blob whose provenance is lost the moment its session ages out.

The failure modes follow from that:

  • Workspace TTL racing artifact upload. The Gitpod data-loss case [9] is the archetype: the workspace timeout fires before the final sync completes, and the deliverable is gone. Any artifact path that depends on workspace storage is exposed to the same race.
  • Unbounded artifact storage. With no retention policy, generated artifacts accumulate. At per-user scale that is a cost leak; at multi-tenant scale it is also a compliance exposure, because every retained artifact is data about a user that erasure must reach.
  • Artifact without attribution. An artifact with no lineage cannot be reproduced, audited, or selectively deleted under right-to-erasure. It is dead weight that is also a liability.
  • Client-server contract drift. Typed artifact taxonomies that evolve independently of client renderers produce user-facing "unknown artifact type" errors.

Storage substrate: where cold-start trades against durability

The PVC is only one substrate, and the same workspace contract can sit on top of several with very different cold-start, durability, and blast-radius properties. The substrate choice is a design axis in its own right, and it is the one that determines whether branching and snapshotting are cheap or ruinous.

  • PVC (ReadWriteOnce, zone-scoped). Simple, zone-scoped, slow teardown race, no native snapshot. Covered above.
  • Copy-on-write overlay (overlayfs, btrfs, ZFS). A base image is shared read-only across sessions; per-session writes go to an overlay. Cold start is fast because the base is already cached on the node, and recovery is fast because the overlay is small. The cost is node affinity: the node must hold the base, so cross-node migration needs lazy fetching and a reschedule loses the locality advantage.
  • Content-addressed snapshots. The workspace is checkpointed to content-addressed storage (think restic or zfs send to object storage). Snapshots compose with deduplication: identical files across sessions share storage. Firecracker snapshots take this to the microVM level, serializing the entire VM (memory plus disk) to object storage and restoring in tens to hundreds of milliseconds [22]. REAP prefetching reports 1.04 to 9.7 times invocation speedup over baseline snapshots, 3.7 times on average, which is the concrete number to size capacity against [23].
  • Lazy-clone volumes. The volume presents immediately and blocks only on page fetches for files not yet pulled. Depot, Namespace Labs, and the containerd-native Nydus/stargz-snapshotter report cold-start in the hundreds of milliseconds even for multi-gigabyte base images [24] [25]. The counterpoint is workload sensitivity: Microsoft's analysis of a 14 GB Python/CUDA stack documents it ballooning to 900 seconds when FUSE-streamed, so sub-second claims are highly workload-dependent [26].
  • Virtualized filesystems (FUSE, object-store-as-FS). s3fs, gcsfuse, or a custom FUSE layer presents object storage as a POSIX filesystem: durable by construction, with per-call latency set by the object store. A strong fit for read-heavy, write-light agents.
  • Ephemeral disk plus git as state. No durable workspace at all. Each session clones fresh, works on a branch, and pushes on completion; the filesystem is scratch and the git host holds the durable state. Maximum operational simplicity, no workspace-persistence problem, but nothing survives between turns unless it is committed.

The four-axis trade-off

graph LR
    classDef fast fill:#d5f5e3,stroke:#27ae60
    classDef slow fill:#f9d6d6,stroke:#c0392b

    subgraph "Cold start"
        A1[Firecracker snapshot: tens of ms]
        A2[Lazy clone: hundreds of ms]
        A3[Overlay CoW: seconds]
        A4[PVC mount: seconds to minutes]
        A5[Git clone from scratch: minutes]
    end

    class A1,A2 fast
    class A4,A5 slow

Cold-start cost, durability guarantee, blast radius on node failure, and snapshot-frequency cost form a four-way trade-off, and no substrate wins on all four. Fast cold-start tends to come from node-local caching, which is exactly what increases blast radius on node failure; strong durability tends to come from object storage, which is what raises per-call latency. The choice is a deliberate placement on that surface, not a default.

Concrete failure modes:

  • Node-pinned state. CoW overlays tie a session to the node holding the base; rescheduling discards the advantage.
  • Snapshot pipeline lag. If the snapshot cadence is five minutes and the pod dies three minutes after the last one, three minutes of work is lost regardless of how good the substrate is.
  • Object-store consistency. FUSE-backed filesystems inherit the store's consistency model. S3's strong read-after-write is recent; older integrations still assume eventual consistency and can read stale state.
  • Cost of full snapshots. Firecracker serializes memory, so a 4 GB VM yields a 4 GB snapshot. At session scale the storage cost is non-trivial.
  • Snapshot memory-state leakage. This is the one that turns a performance optimization into a security incident. Restoring the same snapshot across tenants leaks in-memory secrets and PRNG state, documented explicitly in Firecracker's snapshot-support documentation [22]. A snapshot captures memory, and memory holds whatever the previous tenant left there. Multi-tenant snapshot reuse therefore requires per-tenant zeroing or re-keying; reusing a warm snapshot across a tenant boundary without it is a cross-tenant leak wearing the costume of a cold-start win.

Long-term memory: retrieval that can poison or leak

Sessions are bounded by context compaction. Agents that outlast sessions need durable, queryable memory the harness can load into context. This is a separate system from the session log: the log is append-only intent, memory is curated and retrievable. It is also the first of the two stores genuinely co-owned with identity, because every memory record is data about a specific user or tenant and must be partitioned as such.

graph TB
    classDef short fill:#d6eaf8,stroke:#2980b9
    classDef long fill:#fdebd0,stroke:#e67e22
    classDef external fill:#d5f5e3,stroke:#27ae60

    subgraph "Short-term"
        ST[Session context<br/>within one conversation]
    end

    subgraph "Long-term"
        EM[Episodic memory<br/>past interactions]
        SM[Semantic memory<br/>facts about user / domain]
        PM[Procedural memory<br/>learned workflows]
    end

    subgraph "External knowledge"
        RAG[RAG indexes<br/>documentation, tickets, code]
        KG[Knowledge graphs<br/>entities, relationships]
    end

    class ST short
    class EM,SM,PM long
    class RAG,KG external

The implementation options span vector store plus embedding retrieval (Pinecone [27], pgvector [28], Chroma [29], among others), opinionated memory frameworks layered on top (Letta [30], the successor to MemGPT [31]; Zep [32]; Mem0 [33]), knowledge graphs (entities and relationships, better for compositional queries but harder to populate automatically), RAG over existing corpora, and curated pinned facts (a small high-signal file per user or project). The framework benchmark numbers are contested enough to be worth distrusting: Mem0 reports a 26 percent LLM-as-Judge gain over OpenAI memory on LOCOMO [34] with large latency and token-cost reductions [33]; Letta's counter-benchmark reports a higher LoCoMo score with disputed methodology [35]; Zep reports its temporal knowledge graph leading on a different benchmark [32]. Treat cross-vendor memory benchmarks as contested until independently reproduced.

The mechanism that makes all of this risky is the same one that makes it useful: retrieved content is injected straight into the model's context, where it is indistinguishable from instructions the user or harness put there. That single property generates the two failure modes that matter most.

  • Retrieval poisoning. A misretrieved document, or one written into memory by an earlier turn, lands in the prompt and shifts the agent's behavior in ways that are hard to debug, because the trace shows retrieval succeeding. The injected text is not an error; it is a correct retrieval of wrong content. A sharper variant is memory injection: because the agent writes memory based on user messages, a malicious user can craft messages that plant persistent misinformation, the prompt-injection problem moved into the memory store, where it survives across sessions.
  • Cross-tenant bleed-over. A top-k similarity query against a shared vector store without an enforced tenant filter can return another tenant's data, and that is a data-breach-class failure, not a relevance bug. The defense has to live in the index, not in application code: partition at write time (per-tenant namespaces, pgvector row-level security or separate schemas, logical per-tenant indexes) so a query physically cannot reach another tenant's vectors. Attributing every chunk and filtering after the query is the weaker option, because filter-at-read-time is one SQL bug away from a leak, whereas partition-at-write-time fails closed. This is where the persistence fabric consumes the identity fabric most directly: tenancy is sourced from identity, and the partitioning decision enforces it.

Two further trade-offs shape the design without being failure modes: changing the embedding model invalidates the existing index and forces an expensive reindex, and memory must be either loaded ambiently into every prompt (simple, expensive) or exposed as a tool the agent queries on demand (cheaper, scales better, requires the agent to know when to ask).

Audit, compliance, and data residency

Audit is the second store co-owned with identity, and the co-ownership is total: a tamper-evident record is worthless if it cannot say who did the thing it attests to. Every audit entry carries the (user_id, tenant_id, agent_id, workload_id) tuple resolved by the identity fabric, and right-to-erasure cascades are graph traversals over the identity-ownership graph. Without that graph, "delete all of Alice's data" is a best-effort search rather than a verifiable operation, because nothing tells you where all of Alice's data is.

Agents handle data, and in regulated contexts (healthcare, finance, EU users, enterprise customers under SOC 2 or ISO 27001 commitments) how that data is handled has teeth: GDPR (right to erasure, data minimization, lawful basis, EU residency, applying to both session logs and memory), the EU AI Act (GPAI obligations enforceable from August 2025, with the GPAI Code of Practice as the operational artifact [36] [37]), the NIST AI RMF and its GenAI Profile plus the CSA Agentic AI Profile that covers the agent-specific gaps [38] [39] [40], ISO 42001 [41], SOC 2 Type II, and HIPAA.

The controls are largely mechanical, append-only logs with cryptographic chaining for tamper-evidence, PII redaction applied before storage, per-region deployment that keeps EU data in EU infrastructure end-to-end, right-to-erasure pipelines, data processing agreements with every external processor, and multi-year audit-log retention with archival to cold storage. The two places where the mechanics fight each other are worth stating precisely.

  • Tamper-evidence versus erasure. Cryptographic chaining (a Merkle log, a transparency log) makes a record tamper-evident by linking each entry to the previous one. That same linkage makes deletion break the chain: remove a link and you cannot prove the chain is intact. The standard reconciliation is a tombstone plus a signed proof-of-deletion, the original payload is purged while the chain integrity is preserved over the tombstone, so erasure and tamper-evidence coexist instead of canceling each other out. Most event-log implementations are append-only but not tamper-evident; the cryptographic chaining is the part that creates this tension and the part most often skipped.
  • Erasure versus durability everywhere else. A right-to-erasure request has to cascade across every store that holds the user's data, sessions, memory, artifacts, and traces. That is straightforward for a session store with a clean ownership graph and painful for two specific cases: framework-managed memory that may not expose a delete API at all, and trained models, where the user's data is effectively baked in and "deletion" is not a record operation. The training-data question compounds this: using session logs to fine-tune requires a lawful basis and usually explicit opt-in, and fine-tuning on EU traces in a US-region cluster is a cross-border transfer problem distinct from inference residency that has to be assessed on its own.

The recurring trade-off underneath all of it is completeness versus cost and privacy. Capturing every prompt, response, tool call, and intermediate state with multi-year retention is both expensive and in direct tension with data minimization, so most production systems capture a structured subset, the decisions, not the deliberation, which is a defensible compromise only when the subset is chosen deliberately rather than by accident.

Where this leaves the human

The thread running through why Latere exists is that the most important intelligence in an autonomous system is the one you cannot see: the person who set the direction and drew the boundaries. The persistence fabric is where that authority becomes durable and inspectable. A session log that records intent and result is what lets a person see what an agent actually did. A workspace policy that is honest about its gaps is what keeps a person from trusting recovery that will not happen. A memory store partitioned by tenant is what keeps one person's context out of another's session. A tamper-evident, identity-attributed audit log with a working erasure cascade is what makes the agent's history something a person can both rely on and revoke.

The agent runs at full speed; the record of what it did, and the authority to read, branch, and delete that record, stays with a person. That is the same bet the rest of the platform makes, applied to the one thing every agent leaves behind: state.

Latere runs persistence as internal substrate rather than a separately marketed product; the hub is honest about that, and about the durable-state seams, long-term memory and a unified audit stream among them, that are still open.

References

[1] "AI Agent Reliability Report 2025." LLM agent retry rate 15–30%. fast.io

[2] S. Ewen, G. van Dongen, I. Shilman. "Durable AI Loops: Fault Tolerance across Frameworks." Restate, 2025. Restate

[3] C. Davis. "Durable Execution meets AI." Temporal, 2025. Temporal

[4] C. Poly. "Durable Execution: The Key to Harnessing AI Agents in Production." Inngest, 2026. Inngest

[5] WorkOS. "Maxim Fateev on Durable Execution for AI Agents." 2025. workos.com

[6] Y. Zhou et al. "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering." 2026. arXiv:2604.08224

[7] Rack2Cloud. "Kubernetes Day 2 Failures: 5 Incidents & the Metrics That Predict Them." 2026. Rack2Cloud

[8] Kubernetes Issue #121436. "PVC Binding Prevents Pod Rescheduling." GitHub

[9] Gitpod Issue #9544. "Data loss when workspace timeout fires before sync completes." 2022. GitHub

[10] Karpenter Issue #2777. "PVC Zone Injection Prevents Pod Rescheduling." 2026. GitHub

[11] Replit. "Inside Replit's Snapshot Engine." December 2025. blog.replit.com

[12] Replit. "Finding and Solving Memory Leaks." Replit Blog

[13] Anthropic. "Claude Code: Checkpointing and Session Forks." Claude Code Docs. code.claude.com

[14] Cursor. "Checkpoints in the Agent Workflow." 2025. stevekinney.com notes

[15] Replit. "Checkpoints and Rollbacks." Replit Docs

[16] LangChain. "LangGraph Time Travel and Branching." 2025. LangChain Docs

[17] OpenAI Codex CLI. Conversation rewind (Esc) ships today; a code-reverting rewind is proposed in Issue #11626, 2026. GitHub

[18] Replit. "Replit Agent." Replit

[19] OpenHands. "Runtime Architecture and File Operations." docs.openhands.dev

[20] OpenAI. "Responses API, Structured Outputs and Attached Artifacts." 2025. platform.openai.com

[21] E2B. "Secure Sandboxes for AI Code Execution." e2b.dev

[22] AWS. "Firecracker Snapshotting." Firecracker Docs

[23] D. Ustiugov et al. "Benchmarking, Analysis, and Optimization of Serverless Function Snapshots (REAP)." ASPLOS 2021. arXiv:2101.09355

[24] Depot. "Lazy Container Pulls with Stargz and eStargz." 2025. Depot

[25] Nydus / stargz-snapshotter. "Lazy-Load OCI Images (containerd)." GitHub; GitHub

[26] Microsoft. "Dissecting LLM Container Cold-Start: Where the Time Actually Goes." 2025. techcommunity.microsoft.com

[27] Pinecone. "Managed Vector Database for AI." pinecone.io

[28] pgvector. "Open-source Vector Similarity Search for Postgres." github.com/pgvector/pgvector

[29] Chroma. "Open-source Embedding Database." trychroma.com

[30] Letta. "Stateful Agents with Memory as a First-Class Primitive." letta.com

[31] C. Packer et al. "MemGPT: Towards LLMs as Operating Systems." 2023. arXiv:2310.08560

[32] Zep. "Temporal Knowledge Graph for Agent Memory." getzep.com

[33] "Mem0: Memory for the AI Era." April 2025. arXiv:2504.19413

[34] Snap Research. "LOCOMO: Long-Term Conversational Memory Benchmark." snap-research.github.io

[35] Letta. "Benchmarking AI Agent Memory." 2025. letta.com

[36] European Parliament. "EU Artificial Intelligence Act." 2024. artificialintelligenceact.eu

[37] European Commission. "GPAI Code of Practice." July 2025. code-of-practice.ai; EU Digital Strategy

[38] NIST. "AI Risk Management Framework (AI RMF 1.0)." 2023. NIST

[39] NIST. "AI 600-1, Generative AI Profile." July 2024. NIST

[40] Cloud Security Alliance. "Agentic AI Profile for NIST AI RMF v1." 2025. CSA Labs

[41] ISO. "ISO/IEC 42001:2023, AI Management Systems." ISO