Batch Bankruptcy: Why Background Agents Starve Your Real-Time Users

The fastest way to degrade a healthy customer-facing AI feature is to ship a successful background agent right next to it. For six months, your interactive copilot or support assistant runs with predictable time-to-first-token (TTFT) and a clean error budget. Then another squad launches an asynchronous fleet of repository-sweep agents, bulk document extractors, or nightly evaluation jobs, points them at the same model endpoint, and your interactive latency SLO collapses . Nothing changed in the interactive service’s code. What changed is the arrival distribution: human clicks follow a smooth diurnal curve, whereas background loops arrive as synchronized, high-concurrency walls of long-context requests that starve live users of tokens, compute, and cache capacity.

To fix the contention, you first have to separate two distinct physical failure modes that teams routinely conflate: commercial API rate-limit exhaustion and self-hosted GPU scheduler saturation. When you consume commercial frontier APIs, tokens-per-minute (TPM), requests-per-minute (RPM), and concurrent token budgets are enforced at the vendor organization, project, or workspace boundary. Handing a background agent fleet its own API key inside the same uncapped project or workspace provides zero isolation. When a top-of-the-hour cron job dispatches 500 multi-turn agents against a shared quota pool, they drain the token bucket in seconds, triggering upstream 429 Too Many Requests responses and client retry storms that put paying interactive users at the back of an exponential-backoff queue.

On self-hosted GPU clusters running vLLM or SGLang, the failure mechanism is even more brutal because it happens inside GPU memory and the continuous-batching scheduler. Interactive traffic consists of short incremental turns that depend on fast prefill and steady decode iterations (inter-token latency). Background extraction and codebase-indexing jobs dump 40,000-to-100,000-token prompts into the same worker pool. Those massive prefill chunks monopolize tensor-core compute during scheduling iterations, stalling in-flight decode steps for live users, while their footprint in high-bandwidth memory (HBM) evicts the active prefix KV-cache blocks that keep interactive turns cheap and fast.

Preventing batch bankruptcy requires a two-lane inference admission contract enforced before a request ever reaches the model. For commercial APIs, either isolate background workloads into a dedicated vendor sub-organization, project, or workspace with an explicit hard rate-limit cap where the provider supports it, or route every internal caller sharing that vendor quota pool through a single inference gateway that enforces hard per-lane token-bucket caps. Critically, the gateway’s admission accounting must mirror the upstream provider’s exact limiter and replenishment rules: deduct both counted input tokens and reserved max_output_tokens on admission, and refund unused output tokens on completion only when the upstream provider itself reconciles against actual completion length (retaining the full max_output_tokens charge on providers that meter rate limits against requested output allowance), guaranteeing that background traffic can never touch the 35% to 50% of TPM and RPM reserved for interactive users. For self-hosted clusters, isolate interactive and background workloads onto separate GPU worker pools, or, when using disaggregated prefill and decode architectures, partition or enforce strict priority preemption and compute/HBM reservations across both the prefill pool and the decode pool so batch traffic cannot stall interactive TTFT or decode throughput.

The second half of the contract is replacing wall-clock cron dispatch with utilization-paced pull scheduling. Never trigger 5,000 background agent runs at 00:00 UTC or on the fifteen-minute mark. Place background work into a priority pull queue whose dispatcher leases jobs only when the gateway reports unused capacity in the background token bucket (for commercial APIs) or when scheduler queue depth and KV-cache occupancy sit below target thresholds (for self-hosted pools), adding per-worker jitter so multi-turn continuations do not re-synchronize across turns. Smooth dispatch turns a destructive spike into flat, zero-contention background utilization that harvests idle capacity between interactive peaks.

Finally, wire a three-rung load-shedding breaker into your agent reliability contract driven directly by interactive SLO telemetry. When interactive p95 TTFT or the upstream 429 rate crosses its warning threshold for sixty seconds, Rung 1 pauses new pull-queue leases at the gateway. If pressure persists, Rung 2 signals already in-flight background orchestrators to cap tool-call fan-out, checkpoint their state after the current turn, and yield the worker. Rung 3 diverts non-urgent offline workloads (nightly evaluation suites, bulk backfills, and historical classification) off synchronous endpoints entirely and submits them to asynchronous 24-hour batch APIs, cutting token spend by roughly 50% while removing the load from your real-time quota pool. Audit your inference gateway and vendor organization structure this week: if a background cron job went into a runaway retry loop at 2:00 PM today, what hard mechanism would stop it from taking down your live customer traffic?