How Google AX and Agent Substrate Resume a Suspended Agent

On 21 Sep 2026, Hacker News ranked “AX – Google’s Open Agentic Orchestrator” near the top of the front page (666 points, ~300 comments), pointing readers at agentexecutor.io and the open google/ax repo. The thread made one concrete question loud: how does a Google AX resume suspended agent path actually work? The short answer is that there are two resume paths, and they are not the same command.

AX is the declarative orchestrator. Agent Substrate is the sandbox AX schedules tasks onto. Confusing them produces bad ops diagrams: an ax resume on a Task is not the Substrate HTTP wake of a STATUS_SUSPENDED actor, and vice versa. This post separates those planes with what the current primary docs say — and what they no longer say.

AX is the orchestrator, Agent Substrate is the sandbox

The current AX README states the split plainly: AX is a high-throughput declarative orchestrator that runs on top of Agent Substrate for sandboxed execution. You declare agentic work as ax.io/v1alpha1 manifests — today that means Task, Workspace, and Model — and apply them with ax apply. Substrate must already be running in the cluster; AX expects its Control API in the ate-system namespace.

That layering matters for resume. AX owns the Task control plane (phases, conditions, CLI verbs). Substrate owns actor state machines, checkpoints, worker assignment, and the request path that can wake a hibernated actor. Drawing one blob labeled “AX / Substrate” hides which command you are actually calling.

License is Apache-2.0. The README also warns that AX is in heavy development and expects major breaking changes before a stable release. Treat contracts below as documented-today, not frozen forever.

Two resume paths, and they are not the same command

Path one is the AX control plane. From the README quick start and CLI reference:

ax suspend task task123   # checkpoint and pause
ax resume task task123    # pick up where it left off

Those verbs talk to the AX API (gRPC). They checkpoint and pause a Task, then pick it up again. They are not an HTTP request into the actor’s name.

Path two is the Substrate request path. Substrate’s suspend docs say the next request triggers a resume onto a worker. Resume is end-to-end collaboration across six components and two data stores on a single HTTP request: client HTTP, L7 proxy, ExtProc, ResumeActor (ateapi workflow), restore (atelet / snapshot store), then forward to the worker. ExtProc calls ResumeActor on every request — warm or cold — so routing never depends on a stale worker IP cache.

A closed, unmerged side note exists for completeness only: PR 330 (closed 2026-09-20, not merged) explored an Antigravity conversation continue via ax exec --resume with empty input / chat("") on a STATE_PENDING conversation. That is neither ax suspend nor the Substrate HTTP wake. Do not treat it as a third current path.

One HTTP request wakes a SUSPENDED actor

Substrate actors sit in exactly one of eight statuses stored in Redis: STATUS_UNSPECIFIED, STATUS_RESUMING, STATUS_RUNNING, STATUS_SUSPENDING, STATUS_SUSPENDED, STATUS_PAUSING, STATUS_PAUSED, and STATUS_CRASHED. Suspend uploads an external snapshot and frees the node entirely; pause keeps a local snapshot on the node VM. Both free the worker.

On the cold path, a client resolves the actor DNS name to the atenet router, not a worker IP. The L7 proxy hands headers to ExtProc. ExtProc parses (atespace, actor_name) and calls ResumeActor. ateapi loads the actor, assigns an idle eligible worker, marks the actor RESUMING, asks atelet to restore from object storage (or a local snapshot), then finalizes to RUNNING. ExtProc mutates :authority to the worker pod IP and the proxy forwards the original HTTP request.

On the warm path the same ResumeActor call is a Redis read that returns the current ateom_pod_ip — no restore. Developers care because density depends on parking idle actors without breaking the next inbound call: the wake is request-driven, not a separate “start actor” CLI on the Substrate side of this flow.

What ax suspend and ax resume do

On the AX side, the README table lists “pause an idle agent and pick up exactly where it left off” as the job of ax suspend / ax resume. The DESIGN API reference exposes matching RPCs: SuspendTask (“checkpoint actor state and pause the task”) and ResumeTask (“resume a suspended task”).

Current docs/concepts.md on main describes the status view operators watch. status.phase is a one-word summary — examples include Running, Suspended, Failed, Terminating. Conditions carry detail: WorkspaceReady stays True after setup; Ready is True when the task is running and workspaces are ready. Suspending sets Ready to False with reason TaskSuspended; resume sets Ready back.

Automatic idle suspend is still roadmap language: “Idleness Detection and Automatic Suspension for Density” would trigger SuspendActor when monitoring decides a task is idle. That is planned, not documented as shipping behavior today. Manual ax suspend is what the README shows.

Task, Workspace, and Model, and the Gateway current docs dropped

Current main concepts are three kinds only:

• Task — smallest unit of isolated execution: image, command, compute, env, and Workspace refs under spec.workspaces.
• Workspace — populates the filesystem and tool landscape (Git repos, MCP servers/registries, skill registries). Bind once; the runner materializes it before the command starts. If you are wiring MCP into that landscape, the patterns in the MCP gateway architecture diagram guide are the sibling control-plane question — Workspace declares what the agent can call; a gateway (when you add one outside AX) governs how those calls are auth’d and rate-limited.
• Model — named provider config plus the Kubernetes secret for credentials, not the model weights themselves.

Compare that to v0.3.0 concepts: Task also carried a Gateway reference; there was a GatewayReady condition; and Gateway itself was a first-class concept — listeners plus an egress allowlist of hosts and ports. agentexecutor.io still markets Gateway (“network policies,” allowlist, credential injection). Current main concepts.md omits Gateway entirely, and the main Task definition dropped the Gateway ref. Say that plainly on any diagram: Gateway is historical (v0.3.0) and marketing-site surface, not a current main type.

The v0.3.0 Redis Streams queue is not the current design

AX v0.3.0 DESIGN.md said AX keeps state in Redis and uses Redis Streams as the work queue between the API server and a pool of controllers (ax-controller workers consuming via XREADGROUP). Current main DESIGN.md dropped that shape. Main stores state in Redis and reconciles directly with Agent Substrate under fine-grained distributed locks. The ax-server role on main is described as validating manifests, managing locks, reconciling with Substrate, and persisting to Redis — not publishing stream events for a separate controller fleet.

This is an AX design change, not an Agent Substrate v0.3.0 redesign. Public Substrate tags seen for this post are v0.2.0, v0.1.0, and v0.0.0 only. Do not annotate Substrate boxes with “v0.3.0 Redis Streams.”

Treat "billions of tasks" as a vendor claim

The README says AX is built to run “billions of autonomous agent workloads” / “billions of tasks per cluster.” agentexecutor.io echoes “Scales up to billions of tasks” and “billions of concurrent agent sessions.” That wording is vendor marketing. No independent public benchmark was found for this draft. Hedge it on slides: density and fast suspend/resume are the architectural bet; “billions” is aspirational copy, not a measured result you should cite as fact.

What is safe to rely on today

• AX schedules Tasks onto Agent Substrate sandboxes; keep the two systems separate in design docs.
• ax suspend / ax resume are the documented Task control-plane verbs; phase/condition semantics live in current concepts.
• A SUSPENDED Substrate actor wakes on the next HTTP request through L7 proxy → ExtProc → ResumeActor → restore → worker; ExtProc calls ResumeActor every request.
• Current main concepts: Task, Workspace, Model. Gateway and Redis Streams belong to v0.3.0 (and Gateway still on the marketing site), not current main.
• README: heavy development, breaking changes likely. CONTRIBUTING currently welcomes external PRs — do not repeat older “PRs paused” claims.
• Automatic idle SuspendActor is roadmap, not current shipping behavior.
• Apache-2.0.

FAQ

Is ax resume the same as the HTTP wake?

No. ax resume is an AX control-plane verb on a Task. It talks to the AX API over gRPC and asks the control plane to resume a suspended task. The Agent Substrate HTTP wake is different: the next client HTTP request to a SUSPENDED actor goes through the L7 proxy and ExtProc, which calls ResumeActor, restores the checkpoint, and forwards the request to a worker. Same idea — bring idle work back — different command, different plane, different trigger.

Is Gateway still in current docs?

Not on current main. docs/concepts.md on main defines Task, Workspace, and Model only. Gateway appears in the v0.3.0 concepts docs (egress allowlist and listeners) and still shows up on agentexecutor.io. Current main Task definitions also dropped the Gateway reference that v0.3.0 Task carried. Do not draw Gateway as a current AX type unless you are documenting a historical or marketing surface.

Was Redis Streams an Agent Substrate v0.3.0 redesign?

No. Redis Streams as the work queue between the AX API server and controllers is an AX v0.3.0 DESIGN.md detail. Current AX main DESIGN.md dropped that path and describes Redis plus direct reconcile with Agent Substrate under fine-grained distributed locks. Substrate releases seen publicly are v0.2.0, v0.1.0, and v0.0.0 — there is no Agent Substrate v0.3.0 release tied to that queue change.

Conclusion

Google AX resume of a suspended agent is two cooperating stories: an orchestrator verb that checkpoints a Task, and a Substrate request path that restores a SUSPENDED actor when HTTP arrives. Keep AX’s Task / Workspace / Model model on the control-plane tier, keep the six-component wake sequence on the Substrate tier, mark Gateway and Redis Streams as v0.3.0-era (plus marketing) unless you are pinning an old tag, and treat “billions of tasks” as vendor wording until someone publishes measurements.