A user clicks “Run agent”, your backend receives a normal HTTP request, and the agent starts doing what agents do: calling tools, reading documents, querying APIs, waiting for a human approval, retrying a flaky search, and generating a long report. Two minutes later, your load balancer returns 504 Gateway Timeout. The user sees an error. The agent, depending on where it is running, may still be alive. It may still be spending tokens. It may have already sent an email, created a ticket, or updated a record. Then the user clicks retry. Now you may have two agents doing overlapping work, with no clean way to know which one is authoritative. That is the core problem: An HTTP request is a short-lived delivery mechanism. An AI agent run is often a long-lived state machine. When the agent outlives the request, the request/response model stops being the right abstraction. You need run IDs, durable state, queues, idempotency, cancellation, progress events, and a way to resume after failure. This article is about what breaks when an AI agent runs longer than the HTTP request that started it — and how to design systems that survive that reality. TL;DR HTTP requests are bounded by clients, proxies, gateways, and platform timeouts. AI agents often run for seconds, minutes, hours, or even days. If an agent can outlive the request, do not treat the request as the job. Create an agent run resource, return 202 Accepted, and execute the work asynchronously. Use idempotency keys to prevent duplicate runs. Store agent state durably, not just in process memory. Use SSE/WebSockets for progress, but keep the run state as the source of truth. Treat tool calls as side effects that need idempotency, retries, and sometimes compensation. Cancellation must be explicit and cooperative. For long human pauses or multi-step workflows, prefer durable execution patterns. 📋 Table of Contents The real mismatch 1. The Gateway Times Out, but the Agent Keeps Spending Tokens 2. The Client Retries, and Now You Have Two Agents 3. In-Memory State Dies on Deploy, Restart, or Scale-In 4. Streaming Is a Delivery Channel, Not a Source of Truth 5. Tool Calls Turn Agent Runs Into Distributed Transactions 6. Cancellation Has to Be a Protocol, Not a Hope 7. Human Pauses Make “Long-Running” Look Cute 8. Auth Tokens Expire While the Agent Is Still Thinking 9. Choosing the Right Execution Model for Agent Workloads 10. The Production Shape I’d Use A practical checklist before shipping long-running agents The real mismatch A typical HTTP request has a simple lifecycle: client connects client sends request server processes server responds connection closes An AI agent run has a much messier lifecycle: queued started planning waiting for model waiting for tool waiting for approval retrying streaming progress completed / failed / cancelled Those two lifecycles do not line up. HTTP assumes the server can finish quickly enough for the client to stay interested. Agent work often cannot make that promise. The agent may be waiting on: a slow model response a tool call to an external API a browser automation step a document parser a code execution sandbox a human approval a scheduled follow-up a rate limit cooldown a multi-step plan that only becomes clear halfway through The first mistake is pretending the HTTP request is the agent. It is not. The request is only the trigger. Once you accept that, the architecture changes. 1. The Gateway Times Out, but the Agent Keeps Spending Tokens Scenario: Your frontend calls POST /agent-runs and waits for the final answer. The agent takes longer than your gateway timeout. The client receives a 504, but the backend worker continues executing. Why it matters: Now you have split-brain behavior. The user thinks the operation failed. The system may still be doing work, consuming tokens, calling APIs, and mutating state. This is especially dangerous when the agent has side effects. A timed-out read-only query is annoying. A timed-out agent that can send messages, create orders, or delete records is an incident waiting to happen. Solution: Do not run long-lived agents synchronously inside the HTTP request. Create a run record, return 202 Accepted, and execute the agent asynchronously. import express from "express"; import crypto from "node:crypto"; const app = express(); app.use(express.json()); app.post("/agent-runs", async (req, res) => { const idempotencyKey = req.get("Idempotency-Key") ?? crypto.randomUUID(); const run = await startAgentRunOnce({ idempotencyKey, input: req.body, }); res .status(202) .setHeader("Location", `/agent-runs/${run.id}`) .json({ runId: run.id, status: run.status, statusUrl: `/agent-runs/${run.id}`, eventsUrl: `/agent-runs/${run.id}/events`, }); }); The client receives a run ID immediately. The actual agent work happens in a worker, queue, or workflow engine. Why this works: The HTTP request is no longer responsible for completing the agent run. It is only responsible for starting it safely. The run becomes a first-class resource: POST /agent-runs → start GET /agent-runs/:id → status GET /agent-runs/:id/events → progress stream POST /agent-runs/:id/cancel → cancel 💡 Practical note: 202 Accepted is the honest response for work that has been received but not completed. It is not a hack. It is the correct HTTP semantics. 2. The Client Retries, and Now You Have Two Agents Scenario: The user does not see a result, so they click “Run” again. Or the mobile app automatically retries after a network drop. Now two agent runs are executing for the same intent. Why it matters: Agents are not pure functions. If the agent can call tools, retrying from scratch can produce different decisions and duplicate side effects. A normal API retry problem becomes much worse with agents because the retry may: generate a different plan call different tools interpret the task differently perform side effects twice leave conflicting artifacts Solution: Require an idempotency key for agent creation. A simple Postgres pattern: CREATE TABLE agent_runs ( id uuid PRIMARY KEY, idempotency_key text NOT NULL UNIQUE, input jsonb NOT NULL, status text NOT NULL, created_at timestamptz NOT NULL DEFAULT now() ); Then insert safely: INSERT INTO agent_runs (id, idempotency_key, input, status) VALUES ($1, $2, $3, 'queued') ON CONFLICT (idempotency_key) DO NOTHING RETURNING id; If no row is returned, fetch the existing run: SELECT id, status FROM agent_runs WHERE idempotency_key = $1; In application code: async function startAgentRunOnce(params: { idempotencyKey: string; input: unknown; }) { const runId = crypto.randomUUID(); const inserted = await db.query( ` INSERT INTO agent_runs (id, idempotency_key, input, status) VALUES ($1, $2, $3, 'queued') ON CONFLICT (idempotency_key) DO NOTHING RETURNING id, status `, [runId, params.idempotencyKey, JSON.stringify(params.input)] ); if (inserted.rows.length > 0) { return inserted.rows[0]; } const existing = await db.query( ` SELECT id, status FROM agent_runs WHERE idempotency_key = $1 `, [params.idempotencyKey] ); return existing.rows[0]; } Why this works: The idempotency key becomes the client’s declaration of intent. If the same intent is submitted again, the system returns the existing run instead of creating a new one. ⚠️ Gotcha: Idempotency keys need a meaningful scope. A key like run-agent is useless. Use something tied to the user, tenant, action, and input hash, or let the client generate a UUID per logical attempt. 3. In-Memory State Dies on Deploy, Restart, or Scale-In Scenario: Your agent keeps its conversation history, current plan, and tool-call progress in process memory. Then you deploy a new version, a container scales in, or the machine restarts. The run disappears. Why it matters: Long-running agents need durable state. If the process can die before the run finishes, the state must live somewhere else. This is not just about crashes. In production, processes restart all the time: deployments autoscaling node draining memory pressure platform maintenance worker restarts after errors If your agent state is only in memory, you do not have a long-running agent. You have a fragile process with amnesia. Solution: Persist the agent run state in a durable store. The exact store can be Postgres, Redis, DynamoDB, or a workflow engine, but the state model should be explicit. A useful run state shape: type AgentRunStatus = | "queued" | "running" | "waiting_for_approval" | "completed" | "failed" | "cancelled"; interface AgentRunState { runId: string; status: AgentRunStatus; input: unknown; cursor?: string; messages: AgentMessage[]; pendingToolCall?: ToolCall; artifacts: Artifact[]; error?: string; updatedAt: string; } More important than the exact fields is the discipline: every meaningful step updates durable state the worker can reload the run after restart the API can answer status questions without touching the worker the frontend can reconnect without losing context Why this works: The worker becomes replaceable. The run state survives independently of any one process. This also makes debugging much easier. When someone asks, “What is the agent doing right now?” you can answer from stored state instead of guessing from logs. 🚨 Production warning: If you cannot answer “What step is this agent on?” without attaching a debugger, your agent is not production-ready. 4. Streaming Is a Delivery Channel, Not a Source of Truth Scenario: You use Server-Sent Events or WebSockets to stream agent progress. The connection drops. The user reloads the page. Now the frontend has missed events and has no reliable way to recover. Why it matters: Streaming is excellent for user experience. It is terrible as the only record of what happened. A stream is ephemeral. It tells you what is happening now, or what happened recently, but it does not by itself answer: what is the current run status? what was the last completed step? which tool calls succeeded? did the run finish while the client was disconnected? can the client safely resume? Solution: Keep durable run state as the source of truth. Use streaming as a notification layer on top of that state. A resilient SSE endpoint should support reconnection: app.get("/agent-runs/:id/events", async (req, res) => { const runId = req.params.id; res.setHeader("Content-Type", "text/event-stream"); res.setHeader("Cache-Control", "no-cache"); res.setHeader("Connection", "keep-alive"); res.flushHeaders(); const lastEventId = Number(req.headers["last-event-id"] ?? "0"); const missedEvents = await eventStore.eventsSince(runId, lastEventId); for (const event of missedEvents) { sendEvent(res, event); } const unsubscribe = eventStore.subscribe(runId, (event) => { sendEvent(res, event); }); const heartbeat = setInterval(() => { res.write(": ping\n\n"); }, 15000); req.on("close", () => { clearInterval(heartbeat); unsubscribe(); }); }); function sendEvent( res: express.Response, event: { id: number; type: string; payload: unknown } ) { res.write(`id: ${event.id}\n`); res.write(`event: ${event.type}\n`); res.write(`data: ${JSON.stringify(event.payload)}\n\n`); } The important part is Last-Event-ID. When the client reconnects, the server can replay missed events instead of pretending the connection never dropped. Why this works: The client can lose the stream without losing the run. The UI becomes a projection of durable state, not the only place where progress exists. 💡 Practical note: Heartbeats are not optional. Many proxies and load balancers close idle connections quietly. A periodic comment line like : ping helps keep the stream alive. 5. Tool Calls Turn Agent Runs Into Distributed Transactions Scenario: Your agent calls three tools: search, create ticket, send email. The email tool succeeds, then the run crashes before the state is saved. The worker restarts and retries. Now the email may be sent again. Why it matters: Agents with tools are not just inference loops. They are distributed systems that take actions. Once an agent can affect external systems, you have the usual distributed-workflow problems: retries duplicates partial failure timeouts compensation auditability But agents make these problems harder because the sequence of actions may not be deterministic. Solution: Make tool execution idempotent wherever possible. Give each meaningful tool call a stable idempotency key derived from the run and the logical operation. async function sendInvoiceEmail(run: AgentRunState, invoiceId: string) { const toolCallKey = `${run.runId}:send_invoice_email:${invoiceId}`; const existing = await externalCallStore.find(toolCallKey); if (existing) { return existing.result; } const result = await emailClient.send({ idempotencyKey: toolCallKey, to: run.input.customerEmail, subject: `Invoice ${invoiceId}`, template: "invoice", }); await externalCallStore.record(toolCallKey, result); return result; } If the downstream API does not support idempotency keys, you still need local deduplication: const alreadyPerformed = await sideEffectLog.exists(toolCallKey); if (alreadyPerformed) { return sideEffectLog.resultFor(toolCallKey); } For destructive or irreversible operations, consider requiring explicit approval or using a two-phase pattern: propose action store proposal wait for approval execute once record result Why this works: You are separating “the agent decided to do something” from “the system actually did it.” That separation gives you a place to enforce safety, retries, and auditing. 🧠 The important part: If a tool call can happen twice, the agent run is not safe to retry unless that tool call is idempotent or guarded. 6. Cancellation Has to Be a Protocol, Not a Hope Scenario: The user clicks “Cancel” while the agent is running. Your API updates a database row. The agent, currently waiting on a model call or external tool, has no idea. Why it matters: Cancellation is easy as a UI concept and hard as an execution concept. If the agent does not check for cancellation, it keeps doing work. If it checks too late, it may perform side effects after the user asked it to stop. Solution: Treat cancellation as a cooperative protocol. First, expose a cancellation endpoint: app.post("/agent-runs/:id/cancel", async (req, res) => { const runId = req.params.id; await agentRunStore.requestCancellation(runId); res.status(202).json({ runId, status: "cancel_requested", }); }); Then make the worker check cancellation at safe boundaries: async function executeAgentRun(runId: string, signal: AbortSignal) { while (!signal.aborted) { const run = await agentRunStore.get(runId); if (!run) return; if (run.status === "cancel_requested") { await agentRunStore.markCancelled(runId, { reason: "user_requested", }); return; } const nextStep = await planNextStep(run); if (!nextStep) { await agentRunStore.markCompleted(runId); return; } await executeStep(runId, nextStep, signal); } } The critical detail is where you check. Good cancellation points: before starting a new model call before executing a tool call after a long external wait before committing a side effect before moving to a new planning stage Bad cancellation strategy: hoping the process gets killed relying only on the client disconnecting checking cancellation only at the very end Why this works: Cancellation becomes part of the run lifecycle instead of an afterthought. ⚠️ Gotcha: Cancellation does not automatically undo side effects. If the agent already sent the email, cancellation may only mean “stop doing more work.” Your system needs to know the difference. 7. Human Pauses Make “Long-Running” Look Cute Scenario: Your agent needs approval before sending a high-risk email. The approval may come in ten seconds, ten hours, or three days. Now your agent is not merely long-running. It is suspended. Why it matters: Many agent systems are designed for “slow API calls,” not for “pause until a human responds.” These are different problems. A slow API call can be handled with timeouts and retries. A human pause requires: durable suspension expiration rules reminders escalation resume semantics audit trail possibly re-authentication If you keep the agent process alive while waiting for a human, you are wasting resources. If you do not persist the pause, you lose the run. Solution: Model waiting states explicitly. await agentRunStore.update(runId, { status: "waiting_for_approval", pendingAction: { type: "send_customer_email", payload: emailDraft, requestedAt: new Date().toISOString(), expiresAt: new Date(Date.now() + 48 * 60 * 60 * 1000).toISOString(), }, }); Then resume when the approval arrives: app.post("/agent-runs/:id/approvals", async (req, res) => { const runId = req.params.id; const { approved, approver } = req.body; await agentRunStore.resolveApproval(runId, { approved, approver, resolvedAt: new Date().toISOString(), }); await queue.enqueue("agent.resume", { runId }); res.status(202).json({ runId, status: "resuming" }); }); This is where durable execution engines start to make sense. They are designed for workflows that can wait for long periods without keeping a process alive. Why this works: The agent run becomes a workflow with explicit pause and resume points. That is far more honest than pretending a human approval is just another fast function call. 💡 Practical note: If your agent can wait for humans, design expiration early. A pending action that lives forever becomes a security and compliance problem. 8. Auth Tokens Expire While the Agent Is Still Thinking Scenario: The agent starts with the user’s access token. Twenty minutes later, it needs to call another API. The token has expired. Why it matters: User sessions and agent lifetimes do not naturally align. A user may: close the browser log out revoke consent switch tenants have a token expire require step-up authentication for a sensitive action Meanwhile, the agent may still be running. If you casually pass the original request token into a long-lived background worker, you create both security and reliability problems. Solution: Decide explicitly what identity the agent runs under. Common patterns: Short-lived user token, narrow task Useful when the run is brief and the token lifetime is sufficient. Risk: the token expires mid-run. Refresh token stored securely Useful when the agent acts on behalf of the user for longer periods. Requirements: secure token storage narrow scopes revocation handling audit logging Service principal with delegated permissions Useful when the agent performs system-level work. Requirements: least privilege per-tenant policy checks explicit authorization mapping Step-up reauthorization for sensitive steps Useful when the agent needs approval for high-risk actions. Example: agent requests permission to refund payment system pauses run user re-authenticates system resumes run with fresh approval A practical authorization check before a sensitive tool call: async function authorizeToolCall(run: AgentRunState, tool: ToolDefinition) { const policy = await policyStore.forRun(run.runId); if (!policy.allows(tool.name)) { throw new Error(`Tool ${tool.name} is not allowed for this run`); } if (tool.requiresRecentUserApproval) { const approval = await approvalStore.latest(run.runId, tool.name); if (!approval || approval.approvedAt < minutesAgo(5)) { await agentRunStore.update(run.runId, { status: "waiting_for_approval", pendingAction: { type: tool.name, payload: tool.input, }, }); throw new ToolCallPaused("Recent approval required"); } } } Why this works: You stop assuming that the original HTTP request’s auth context is valid forever. 🚨 Production warning: Do not solve token expiration by giving agents broad admin tokens “just to keep things moving.” That turns every long-running agent into a privilege-escalation risk. 9. Choosing the Right Execution Model for Agent Workloads Not every agent needs the same architecture. The right choice depends on how long the agent can run, whether it has side effects, and whether humans are involved. Approach Best for Complexity Weakness Synchronous HTTP Very fast, read-only agent calls Low Breaks as soon as work is slow or unreliable Async run + polling Simple background agent jobs Medium Polling can be inefficient; needs run store SSE/WebSocket progress Interactive UX with live updates Medium Connection loss handling is required Queue workers Scalable background execution Medium Retry and state discipline needed Durable workflow engine Multi-step, long-lived, human-in-loop agents High More operational complexity When synchronous HTTP is fine Use synchronous request/response when: the agent usually finishes in a few seconds tool calls are read-only or low-risk there is no human approval losing a run is acceptable retries are safe Example: POST /summarize-text When async run + polling is better Use async runs when: the agent may take tens of seconds or more the client may disconnect you need a stable run ID retries need to be deduplicated you want auditability Example: POST /agent-runs GET /agent-runs/:id When streaming matters Use SSE or WebSockets when: users need live progress tokens or characters are being generated tool activity should be visible you want a responsive UI But keep durable state underneath. When durable execution earns its complexity Use a durable workflow engine when: runs can last hours or days human approvals are common steps need guaranteed retries timers and schedules matter compensation or rollback is needed you need strong execution history This is the category where “agent” starts looking less like a chat endpoint and more like business process automation. 10. The Production Shape I’d Use If I were shipping an AI agent feature that could run longer than a normal HTTP request, I would use this shape: Client ↓ API layer ↓ Run store ↓ Queue / workflow engine ↓ Agent worker ↓ Tool execution layer ↓ Event store / notifications API layer Responsible for: creating runs enforcing idempotency returning run URLs serving status exposing event streams accepting cancellation and approvals Run store Stores: run ID tenant/user input status state cursor pending action error timestamps idempotency key Queue or workflow engine Responsible for: scheduling workers retrying failed executions handling long waits enforcing timeouts preserving execution order where needed Agent worker Responsible for: loading run state calling the model choosing next steps executing tools checkpointing progress checking cancellation emitting events Tool execution layer Responsible for: validating tool input enforcing authorization applying idempotency keys logging side effects rate limiting wrapping external APIs Event store / notifications Responsible for: progress events reconnection support webhooks audit trail UI updates The most important design decision is this: The HTTP request starts and observes the run. It does not own the run. That one mental shift prevents a large class of production pain. A practical checklist before shipping long-running agents Before letting an agent run longer than the HTTP request that started it, I’d want these boxes checked. Lifecycle [ ] Does the run have a unique ID? [ ] Is creation idempotent? [ ] Is the initial response 202 Accepted? [ ] Can the client check status without restarting work? [ ] Can the client reconnect after losing the stream? State [ ] Is run state stored durably? [ ] Can a worker resume after restart? [ ] Are checkpoints saved after meaningful steps? [ ] Is there a clear status model? Side effects [ ] Are tool calls idempotent where possible? [ ] Are destructive actions gated? [ ] Are external calls logged? [ ] Can partial failure be understood and repaired? Cancellation [ ] Can the user request cancellation? [ ] Does the worker check cancellation at safe boundaries? [ ] Are side effects prevented after cancellation where possible? [ ] Is the final cancelled state recorded? Human interaction [ ] Are approvals modeled as explicit run states? [ ] Do pending actions expire? [ ] Can the run resume after a long pause? [ ] Is the approver recorded? Auth and policy [ ] Is the agent’s identity explicit? [ ] Are tokens scoped narrowly? [ ] Are sensitive actions reauthorized? [ ] Are permissions checked per tool call, not just at run start? Observability [ ] Can you see the current step? [ ] Can you trace a run from request to final result? [ ] Are token usage, tool calls, and errors logged? [ ] Can you distinguish user retries from internal retries? The deeper truth is that long-running AI agents are not just “slower APIs.” They are workflows with non-deterministic planning, external side effects, and user expectations. HTTP can trigger them. HTTP can report on them. HTTP can stream progress from them. But once the agent can outlive the request, HTTP should not be the container for the entire execution. The request is the doorway. The agent run is the process. Design them separately, and the system becomes far easier to operate. Design them as the same thing, and every timeout, retry, deploy, and disconnected browser becomes a potential corruption of the run.