Distributed execution substrate for openJiuwen Agent Runtime

English | Chinese

Agent DX (Agent Distributed eXecutor) is a distributed execution substrate for openJiuwen Agent Runtime. Its Agent layer provides Template and AgentBinding management, with stateless Activators starting user Harnesses on demand for HTTP, WebSocket, and SSH access. It also provides a public Environment API and SDK, distributed scheduling, isolated execution, traffic routing, runtime operations, checkpoint recovery, and deployment tooling, while keeping the execution backend replaceable.

🚀 Quick start · 📐 Architecture · 📦 Environment SDK · ⚙️ Deployment · 🔐 Administration · ✅ Test gates

🎯 When to use Agent DX

Your goal	Deployment path	What you need
Add isolated command, file, terminal, and port operations to one agent service	Standalone	One Linux host, separately managed sandboxd, an ADX release package, and Environment networking
Share execution capacity across agent services and machines	Split-host roles	Persistent Redis, one Coordinator deployment, one or more worker deployments, and a reachable Gateway
Run a production-style cluster with repeatable acceptance	Kubernetes	Linux workers, persistent Redis, component certificates, sandboxd on worker nodes, and the ADX Kubernetes E2E profile
Pause, resume, clone, or recover long-running work	Standalone or distributed	A checkpoint-capable runtime plus local, S3-compatible or AFS checkpoint storage
Schedule accelerator workloads	Distributed	GPU/NPU inventory from workers and matching device requests in the Environment specification

🧩 How Agent DX fits your stack

Layer	Owns	Agent DX relationship
Agent framework or application	Prompts, tools, sessions, task logic, and business policy	Calls the public Environment SDK or HTTP API; it does not enter platform scheduling or lifecycle state machines
Agent DX	Authentication, placement, Environment state, routing, recovery, and observability	Provides one distributed execution contract across nodes and execution backends
sandboxd	Runtime creation, isolation, networking, and checkpoint primitives	Runs as an external service on each worker and is accessed through the adxlet runtime driver
Execd inside the Runtime	Commands, files, terminals, ports, activity, and runtime recovery cooperation	Receives data traffic through Ingress and Relay after ownership and generation checks
Redis and shared storage	Authoritative cluster metadata and optional shared checkpoint files	Redis stores control state and discovery; S3 or AFS enables cross-node checkpoint access

🔄 How it works

Step	What happens	Where it lives
1 · Access	A caller authenticates and creates or operates an Environment through one public contract.	gateway/apiserver/, platform/sdk/environment/python/
2 · Place	API Server uses local-first admission or sends the request to Coordinator. Coordinator rotates across Shards; each Shard queues, filters, scores, and reserves a node.	platform/coordinator/, platform/crates/scheduling/
3 · Run	adxlet performs final local admission, serializes the Environment lifecycle, and dispatches to sandboxd or the configured Jiuwenbox driver to create a Runtime with Execd.	platform/adxlet/, third_party/sandboxd/, platform/runtime/execd/
4 · Operate and recover	Ingress routes data to the owning Relay. Versioned ownership fences old Runtimes; pause, resume, snapshots, restart policy, and reconciliation converge state after failures.	gateway/, platform/adxlet/, Redis, checkpoint storage

API Server embeds Ingress by default, and adxlet embeds Relay by default. Both pairs keep the same contracts when configured as separate processes.

📦 Installation

ADX release packages target Linux and contain adx-coordinator, adxlet (embedded Relay by default), adx-apiserver (embedded Ingress by default), the optional standalone adx-ingress and adx-relay binaries, adxctl, the read-only adx-inspect, Execd, the Python Environment SDK, and managed Redis with redis-cli. The debug forwarder is a source-build tool. sandboxd remains independently managed and is pinned by third_party/sandboxd/source.json.

The opt-in Jiuwenbox adapter supports the supervisor and conch classes using independently managed Jiuwenbox/Conch services. Execd uses a separately mounted EROFS artifact. See the runtime acceptance tools for deployment prerequisites and reproducible checks.

mkdir adx-release
tar -xzf adx-release.tar.gz -C adx-release
sudo ./adx-release/install.sh

The installer verifies the manifest, file digests, and host architecture. It installs the release under /opt/adx/releases/<commit>, atomically switches /opt/adx/current, preserves /opt/adx/config, /opt/adx/data, and /opt/adx/run across upgrades, and exposes adxctl through /usr/local/bin.

🔧 Quick start

The default standalone profile starts managed Redis, Coordinator, adxlet with embedded Relay, and API Server with embedded Ingress on one host. Prepare sandboxd, networking, certificates, and the initial administrator key first.

sudo adxctl config init --profile standalone
sudoedit /opt/adx/config/deployment.yaml
sudo adxctl validate
sudo adxctl run

Install the platform-independent administrator wheel on an operator workstation, then create the first tenant Key through the public HTTPS API:

pipx install ./adxadmin-0.1.0rc2-py3-none-any.whl
export ADX_ENDPOINT=https://adx.example.com:8443
export ADX_CA_FILE=$HOME/.config/adx/public-ca.pem
export ADX_ADMIN_TOKEN_FILE=$HOME/.config/adx/admin.key
adxadmin key create --tenant example

See the adxadmin guide for listing, pagination, revocation, JSON output, and the error contract.

In another terminal, install the packaged SDK and point it at the public Gateway:

python3 -m venv /opt/adx-client
/opt/adx-client/bin/python -m pip install /opt/adx/current/sdk/adx_environment-*.whl

export ADX_SERVER_ADDRESS=adx.example.com:8443
export ADX_GATEWAY_ADDRESS=adx.example.com:8443
export ADX_TOKEN="$(cat /secure/path/tenant-api-key)"
export ADX_TLS=1
export ADX_GATEWAY_TLS=1
export ADX_ENVIRONMENT_IMAGE=python:3.12-slim

Create an Environment, execute through Execd, and delete it explicitly:

import os
from adx_environment import Environment

environment = Environment(
    image=os.environ["ADX_ENVIRONMENT_IMAGE"],
    cpu=1000,
    memory=2048,
    name="readme-demo",
)
try:
    result = environment.commands.run("printf 'hello from ADX\\n'")
    print(result.stdout)
finally:
    environment.kill()

The image must be supported by the configured sandboxd and ADX Runtime Environment. See the Environment SDK guide for pause/resume, reusable snapshots, placement, mounts, network configuration, data-plane security, and retry semantics.

🏗️ Deployment options

Topology	adxctl profile	Processes on this host
Standalone with managed Redis	standalone	Redis, Coordinator, adxlet + Relay, API Server + Ingress
Standalone with external Redis	standalone-external-redis	Coordinator, adxlet + Relay, API Server + Ingress
Control host	coordinator	Coordinator; Redis may be managed separately or added to the full YAML
Worker host	node	adxlet + Relay by default
Ingress host	ingress-api	API Server + Ingress by default

Every host has its own /opt/adx/config/deployment.yaml. Cluster members share the Redis URL, namespace, and mTLS trust. Every worker has a unique node_id and reachable control and proxy addresses. Start Redis → Coordinator → workers → API Server. Set proxy_mode: standalone or ingress_mode: standalone only when those components need separate processes.

YAML string values support ${VAR} and ${VAR:-default}. Use adxctl config dump to inspect the fully resolved profile and host overrides. Complete fields and certificates are documented in the adxctl reference, standalone guide, and configuration examples.

📐 Core abstractions

ADX separates stable logical identity from replaceable execution:

Abstraction	Meaning and boundary
AgentBinding	Business identity (tenant, template, version, binding_id) mapped to a stable environment_id; managed through stateless Activators
Environment	Public API and SDK handle presented to applications
Platform Environment	Stable internal identity containing tenant, specification, lifecycle, and desired/observed state
RuntimeProfile	Deployment-owned rootfs, bootstrap and startup variables; configuration rather than a running Environment
Runtime	One concrete backend execution of an Environment on one node; restart or recovery may replace it
Assignment	Authoritative node and device ownership with a generation that fences late old execution
Route / Binding	Versioned ownership published to Ingress and rechecked locally by Relay before forwarding
Restore Point / Snapshot	A restore point keeps the Environment ID; a reusable Snapshot creates a new Environment
Request ID / Operation ID	One logical write identity for retry, deduplication, result lookup, and reconciliation

The fixed control path is Agent/application → Environment SDK/HTTP API → API Server → Coordinator/Shard scheduler or local-first adxlet → runtime driver (sandboxd or Jiuwenbox). Runtime traffic follows Ingress → Relay → Execd and does not enter lifecycle queues. Public lifecycle and error payloads identify the Environment with environmentId. The directory keeps /api/instances and its instance_id query parameter; internal Rust types, RPCs, persistence keys, metrics, and runtime identity use Environment terminology.

Execd can initiate a dedicated bidirectional control stream to adxlet, publishing Ready and checkpoint state while receiving checkpoint commands. Enable it with the node runtime-control configuration; both the node binary and the Execd inside the runtime image must be updated. Without that configuration, runtime cooperation uses HTTP. The contract and component validation describe identity fencing, reconnect and acknowledgements after authoritative result submission.

Checkpoint storage supports local only, S3 with a node cache, and direct AFS filesystem access. Shared publication completes the backend persistence barrier before authoritative metadata is acknowledged; local only retains readable-file semantics without forced writeback. Anonymous recovery points belong to their Environment; active execution inputs remain protected until confirmed cleanup. Coordinator reclaims shared S3/AFS artifacts, while nodes manage local files and cache eviction. Deleting an in-use reusable Snapshot returns SNAPSHOT_IN_USE with the Sandbox list. See the storage contract and validation boundaries.

Component names and abstractions.

✨ Capabilities

• Environment creation, query, deletion, pause, resume, reusable snapshots, and snapshot-based cloning.
• Central and local-first creation with CPU, memory, disk, GPU/NPU device, label, affinity, and preference constraints.
• API Key administrator and tenant identities plus configurable internal mTLS.
• Versioned route publication and synchronized local binding checks.
• Node reconciliation, restart policy, idle deletion/pause, traffic-triggered resume, checkpoint recovery, local/S3/AFS storage adapters, and reference-aware artifact cleanup.
• Prometheus metrics, OpenTelemetry traces, structured component logs, and external Collector integration. Node runtime stdout/stderr is redirected to per-runtime files with terminated-log compression and GC.
• Process deployment with managed or external Redis and Kubernetes end-to-end deployment profiles.

🛠️ Development and test

The root Cargo workspace contains the platform and Agent components. Python packages build independently. Build outputs belong under out/ or a configured external cache.

make help
make rust-check
make rust-test
make agent-test
PYTHONPATH=platform/sdk/environment/python \
  python -m pytest -q -c platform/sdk/environment/pytest.ini \
  platform/sdk/environment/python/tests
make package PYTHON=/path/to/venv/bin/python

Agent FS (AFS), comprising OwnerFs and DistributedFs (DFS), is ADX's optional filesystem subsystem. Default builds, tests and packages exclude its runtime binaries and system dependencies. The public source gate accepts ADX_WITH_AFS=1 to include AFS lint in the same fmt/Clippy/tooling gate without injecting AFS artifacts into the default release. Set ADX_WITH_AFS=1 when running the filesystem checks locally or building an explicit filesystem package:

ADX_WITH_AFS=1 make afs-check
ADX_WITH_AFS=1 make platform-release
python3 build/ci/run.py afs

afs-check uses default OwnerFs and DFS features. The optional all-features/RDMA lint is explicit (ADX_AFS_ALL_FEATURES=1 make afs-lint) and first checks for libibverbs development files.

The subsystem responsibilities and current limits are in architecture, deployment, testing, the AFS roadmap, and the DFS plan. Provenance and dated historical evidence remain under docs/migration/. The packaged configuration examples live under AFS examples. Ordinary release packages do not include them unless ADX_WITH_AFS=1 is set.

Run component and integration suites with python3 build/ci/run.py <suite>. End-to-end gates use installed release artifacts, the public Environment SDK, Redis, Gateway, the control plane, sandboxd, and Execd. Environment requirements and gate definitions are in control-plane CI and the Kubernetes E2E guide.

Buildkite uses independent agent-dx (including SDK, adxadmin and K8s L0), agent-dx-python-sdk, agent-dx-admin, and agent-dx-full-test pipelines. The SDK and admin pipelines always build and check their wheel/sdist; each PyPI upload is an independent explicit, tag-gated option. The Full pipeline consumes explicit base-package and SDK build UUIDs and does not rebuild either candidate. See the Buildkite pipeline contract.

The base-package pipeline builds Linux amd64 and arm64 packages in parallel. Both architectures use the same compilation, assembly, publication, and indexing steps. Tests run on x86; ARM builds and validates artifacts by default, with tests available through ADX_ARM_TESTS=1. ARM builds run in Linux containers on native ARM workers. Artifacts and OBS paths are separated by architecture. Kubernetes L0 currently validates amd64. See the pipeline guide.

The SDK distribution is adx-environment, its Python import is adx_environment, and its CLI is adx-environment. ADX environment variables use the ADX_ prefix, and internal branded HTTP headers use X-ADX-.

📚 Learn more

• Architecture and repository layout
• Agent usage
• Environment API and OpenAPI
• Coordinator autoscaling HTTP
• Data-plane OpenAPI
• Runtime HTTP actions and errors
• Environment Python SDK
• Deployment configuration and examples
• Remote cluster administration and API Key management
• Scheduling, node lifecycle, and route publication
• Checkpoint and snapshot storage
• Runtime stdout/stderr and retention
• Failed Environment GC
• Metrics, logs, and distributed tracing
• Rust coding guidelines
• Release pipelines and package layout
• Brand and architecture assets

License

Agent DX is released under the Apache License 2.0.

Creation throughput, Running/Paused views and tiered resource collection: observability capacity.