When an Agent Times Out After the Write Succeeded

Distributed systems engineers know how to make an HTTP endpoint idempotent when a deterministic caller replays the exact same byte payload with the same client-generated UUID. Autonomous agents in Go break that contract at the worst possible moment: right after the downstream write actually succeeded. Suppose an agent invokes a tool to issue a customer credit, amend a contract record, or reserve inventory; the downstream service commits the transaction, and the network connection times out before the response reaches your tool gateway. A deterministic RPC client retries the identical request body with the same idempotency header. An LLM agent does not. On the next turn, the model reads the timeout error, often refreshes the entity state (seeing version v11 instead of v10), and reformulates the tool call with a brand-new tool_use ID and a slightly reworded free-text memo or reason. Standard middleware that hashes the raw JSON payload or trusts the caller’s request token treats that retry as a new operation and executes the mutation a second time.

Closing that gap requires moving idempotency out of the model’s control and into the tool gateway via a two-tier step lease and fence-before-query reconciliation contract. The first rule is what must never enter your idempotency hash: never let the model supply the idempotency token, and never hash the target entity’s mutable state_version or free-text commentary (memo, reason, notes) into the operation identity. A successful write advances the entity version from v10 to v11, and a language model rewords free-text notes on almost every retry; hashing either field guarantees a key miss on the exact failure mode you are trying to catch. Pass expected_version only as a downstream optimistic-concurrency precondition, and index every side-effecting tool call in the gateway using a two-tier lease key: a primary step lock on (workflow_run_id, plan_step_id, target_entity_id, action_verb) paired with a secondary canonical_business_args digest computed strictly over structured, business-invariant fields (such as amount_minor_units, currency, counterparty_id, and line_item_ids).

When a tool call arrives at the gateway, the primary step lock runs first. If an active (PENDING) or succeeded (COMMITTED) lease already exists for (workflow_run_id, plan_step_id, target_entity_id, action_verb) and the retry’s structured canonical_business_args digest differs from the leased digest, meaning the model changed the dollar amount or counterparty mid-step, the gateway halts the step immediately as CONFLICT rather than guessing which parameters are right. If the prior attempt ended in a definitive downstream business or validation rejection (REJECTED, such as card_declined or an exceeded credit limit where no mutation occurred), a retry with the identical digest returns that stored REJECTED receipt immediately, while a retry with corrected parameters (a new canonical_business_args digest) is permitted to open attempt_epoch + 1. When the structured business fields match an existing PENDING or COMMITTED lease, even if the free-text memo was rewritten, the gateway recognizes the call as a retry of the same logical operation.

Before the gateway dispatches an attempt, it writes a durable PENDING lease record to the session state store . That write-ahead record stores five fields: the leased canonical_business_args digest, the lease state and downstream receipt slot (PENDING, COMMITTED, REJECTED, or ABORTED), the lease creation timestamp, the deterministic downstream idempotency key (step_tuple + attempt_epoch), and the exact serialized downstream request payload from that initial attempt. Storing the original serialized payload is non-negotiable, because strict upstream APIs (such as Stripe) reject idempotency-key replays with an idempotency_error if a single parameter, including a free-text description, differs from the first request.

When a retry with matching canonical_business_args collides with an existing lease, the gateway’s behavior depends on the lease state. If the lease is COMMITTED, the gateway returns ALREADY_APPLIED along with the stored downstream receipt immediately, solving the headline timeout-then-reworded-retry bug without touching the network. If the lease is PENDING, meaning the previous attempt timed out or the gateway crashed mid-flight, your reliability contract forbids lying to the model that the write succeeded before verifying downstream state. The gateway resolves a PENDING lease through one of two deterministic branches. Branch A applies when the downstream API natively honors idempotency keys and lease_age + request_timeout_deadline + clock_skew_margin sits strictly inside the provider’s guaranteed key-retention window (for example, well under 24 hours on Stripe, with a bounded request deadline so an in-flight replay cannot arrive after key eviction and execute a second charge): the gateway replays the stored original serialized payload under step_tuple + attempt_epoch to fetch the committed or rejected receipt, then transitions the lease to COMMITTED or REJECTED. Branch B applies when the retention window cannot be guaranteed, or when the downstream system lacks idempotent replay and the unconfirmed attempt must be aborted before issuing attempt_epoch + 1. Here the ordering is strict: fence first, query second. The gateway first fences attempt_epoch (either by writing an atomic negative tombstone for attempt_epoch in the downstream store, or by waiting until a downstream-enforced request deadline plus clock-skew margin has elapsed) so a slow worker cannot apply attempt_epoch later. Only after that fence is active does the gateway query downstream state to check whether attempt_epoch committed before the fence closed; if it committed, the lease moves to COMMITTED (ALREADY_APPLIED), and only if post-fence verification confirms no write occurred does the gateway mark attempt_epoch as ABORTED and dispatch attempt_epoch + 1. If the downstream system cannot both fence and verify post-fence state, the gateway halts the run as IN_DOUBT and pages an operator.

Test this contract in staging with a single fault-injection drill: proxy a side-effecting tool call so the downstream write commits normally, drop the HTTP response to trigger a gateway timeout, and let the agent retry on the next turn. If your agent issues a second write, fails with a parameter-mismatch error on its reworded memo, or marks a crashed pre-dispatch lease as ALREADY_APPLIED, your idempotency layer was built for deterministic RPC clients, not for an agent.