Separate the Brain, the Hands, and the Session

The default way most teams ship an execution-capable agent is to pack the orchestration loop, the in-memory conversation state, and the shell or code-execution workspace into a single container. It feels clean on a whiteboard: one user session equals one container, and when the agent needs to run a script or inspect a file, it calls exec locally. In production, that coupled container becomes a stateful pet with three compounding failure modes. First, Time-to-First-Token waits on container cold-start, image pulls, and filesystem initialization even when the user’s first turn only requires a model call. Second, a runaway subprocess, memory spike, or kernel OOM inside the sandbox kills the orchestrator process living in the same cgroup and wipes out the unpersisted turn state. Third, scaling concurrent sessions forces you to pay for provisioned CPU and RAM across thousands of idle sandboxes that spend ninety percent of their wall-clock life waiting on network I/O from a model endpoint.

The fix is to split the agent runtime into three separate planes with independent lifecycles: a stateless orchestrator (“the brain”), an externalized session store (“the session”), and late-bound workspace sandboxes (“the hands”). In April 2026, Anthropic’s engineering write-up on scaling Managed Agents documented this exact architectural decoupling in production: by separating the stateless harness from the execution sandbox and pulling session history from an external event log, inference could start before any container was provisioned, cutting p50 Time-to-First-Token by roughly 60% and p95 Time-to-First-Token by more than 90%. You do not need hyperscaler scale to hit the problem this solves; any platform team running fifty concurrent coding, data-analysis, or infrastructure agents hits the same latency and crash-recovery wall.

Plane one is the stateless orchestrator. Implemented as a fleet of lightweight Go worker processes, the orchestrator holds zero local filesystem state, zero in-memory session ownership, and zero customer code. When a turn arrives, any worker replica claims the turn lease, compiles the prompt from the session store, and opens the streaming model request immediately. Because no container boot sits in the critical path of turn one, interactive latency drops to the speed of a database read plus the model’s prefill time. If an orchestrator pod is evicted during a deployment or crashes mid-stream, no committed session state dies with it.

Plane two is the externalized session store, built on the append-only event log rather than local disk or sticky memory. The session store exposes a narrow RPC contract: positional slice reads (getEvents(sessionID, fromSeq, toSeq)) and atomic, sequence-checked event appends. Every state transition in the loop (user input, model response, tool_use intent, tool_result receipt, and sandbox snapshot reference) commits to the log before the next step begins. On crash recovery, a replacement orchestrator worker reads the event slice from sequence zero to the latest committed watermark, reconstructs the compiled context projection, and resumes without re-running or re-billing already-committed turns. If the previous worker died with an in-flight model stream or an uncommitted tool_use intent at the tail of the log, the replacement worker polls the provider’s response ID (if supported) or reconciles the pending tool call through the gateway’s idempotency lease before deciding whether to replay or advance.

Plane three is the late-bound workspace sandbox, and its design has to reconcile two requirements that naive pool designs get wrong: strict credential isolation and multi-turn filesystem continuity. The orchestrator does not allocate a sandbox when a session starts; it claims a pre-warmed, unprivileged container or microVM from a pool only on the first turn that actually invokes a shell or file tool. That sandbox holds no model API keys, no production database credentials, and no ambient service identity ; any external API mutation routes back through the orchestrator’s L7 tool gateway rather than executing raw network writes from inside the shell. Once claimed, however, an agent editing a repository or analyzing a dataset needs the files it wrote in turn three to still be there in turn four. Back the leased workspace with per-turn layered copy-on-write snapshots (L_0, L_1, ...) keyed to the session log’s turn sequence. If the sandbox stays leased across rapid consecutive turns, each completed turn commits a snapshot layer L_t; if the session idles between human replies, the runtime can release the compute container to the destructor queue and re-mount L_t into a fresh container when the next tool call arrives. When a local command fails or crashes midway through turn t, the runtime mounts the last committed layer L_{t-1} and retries cleanly; when a source document first introduced at turn k is revoked, it rolls back to L_{k-1} (or resets to L_0), and when the task completes, the container and its ephemeral layers are destroyed, never dirty-recycled to another tenant.

Operating this three-plane split requires watching two numbers that single-container teams never instrument: warm-pool claim latency at p99 (so bursts do not re-introduce cold-start stalls at the first tool call) and copy-on-write snapshot storage TTLs (so abandoned sessions do not leak block storage indefinitely). Set a hard lease TTL that compacts or purges dormant workspace snapshots after the session retention window closes. The diagnostic question for your platform review takes thirty seconds: if you send SIGKILL to the container running an agent’s local build command right now, does the user lose their session, or does a stateless orchestrator attach a fresh sandbox to the last committed snapshot layer L_{t-1}, reconcile the interrupted step, and keep going?