Skip to content

feat(sdk): add checkpointable Compose integration on current SDK - #127

Draft
hky1999 wants to merge 2 commits into
inclusionAI:developfrom
hky1999:hky1999/sdk-managed-compose
Draft

hky1999 wants to merge 2 commits into
inclusionAI:developfrom
hky1999:hky1999/sdk-managed-compose

Conversation

@hky1999

@hky1999 hky1999 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Harbor currently carries task semantics together with Docker preparation and process recovery plumbing. Move the reusable Docker/Compose lifecycle into the AKernel SDK so Harbor can retain its Environment adapter, model proxy and checkpoint trigger/aligned application state.

Review baseline

Reconstructed directly on inclusionAI/AKernel:develop at 5ded29d; current head 61606ce. The diff contains only the pending managed Compose feature. It retains the newly merged snapshot/lifecycle SDK, command ID/PID identity, conservative creation ownership, cancellation/rollback and fork cleanup contracts. No older snapshot implementation, metadata/lifetime or callback preview APIs are reintroduced. Core backend changes are limited to forwarding validated extra_config through restore and fork.

Changes to review

  • ComposeProject: frozen input preparation, automatic startup-image discovery, explicit start/build/pull/up boundaries, service lookup and JSON descriptors. Original user Dockerfiles and Compose bytes are not rewritten.
  • DockerRuntime and builder/image-bridge: metadata-only OCI validation/registration and the existing Sandboxd shared ROFS lower provider; each sandbox keeps its private managed upper. There is no second layer cache.
  • ServiceHandle / ServiceProcessHandle: exec/transfers/service control and captured process identity. Restore attaches the original container/PID/start-time identity without replaying build/up/agent launch; unknown launch outcomes retain a recovery token. Trees thaw descendants before waiting ancestors to avoid Bash tool status 147.
  • Native default/custom IPv4 bridges, DNS/aliases and userland TCP/UDP mappings; SDK-owned in-sandbox forwarding/SNAT and explicit reconnect after a new outer TAP. A peer-restricted service gateway reuses the existing reverse tunnel and relays HTTP/auth/SSE bytes unchanged.
  • Public documentation, a maintained compose_checkpoint.py example, migration regressions, bridge Go CI and clean-distribution resource/import checks.

Start with component boundaries and integration order, then review _compose.py, _service_process.py, _docker_runtime.py and the bridge helper.

Dependencies and merge boundary

The file-backed upper profile depends on Sandboxd #83. After that merges, the AKernel consumer must pin the merged Sandboxd revision and qualify its ordinary build. AKernel #126 separately exposes the explicit Helm reflink profile. Reflink/durable sync, compatible same-mount storage and Docker 28 packet-socket flags remain operator prerequisites. This PR does not modify the Sandboxd gitlink, runtime release pins, production deployment or Coordinator. Existing Agent DX APIs carry the requests; no Compose-aware Agent DX API is added.

Harbor can consume this SDK source revision through its existing Environment/service adapters. Its delivered branch still pins the earlier prototype; updating that delivery pin and its document requires fresh isolated end-to-end qualification.

Validation on this revision

  • make sdk-check: 459 unit tests + 35 benchmark-driver tests, Ruff and mypy passed; existing upstream ownership/rollback/fork regressions retained.
  • Bridge formatting, vet, Go tests and Linux/amd64 static build passed.
  • Wheel and sdist built and each installed in its own clean environment; declared dependencies, public Compose APIs, packaged guest scripts and CLI smoke checks passed.
  • Existing Harbor native-network branch: 187 related tests passed using this SDK and the previously qualified LiteLLM 1.93.0. A separate exploratory run with LiteLLM 1.106.5 failed seven model-gateway tests; upgrading that dependency is outside this SDK migration.
  • No cluster test or deployment update was performed for this rebased revision. The opt-in two-restore native-network integration regression is included for an owned compatible test instance.

The earlier prototype passed real-Claude original-process C/R and separate-verifier execution on unchanged kv-live-surgery inputs; those results predate this current-develop migration and are not evidence that the new revision/bundle has passed E2E. One prototype checkpoint completed serialization but did not return within 300 seconds; later successful cycles did not determine the cause. Keep this Draft pending review and fresh isolated qualification.

@hky1999 hky1999 changed the title feat(sdk): own managed Docker image preparation and bridge runtime feat(sdk): manage Compose workloads and checkpointable service processes Oct 28, 2026
@hky1999 hky1999 changed the title feat(sdk): manage Compose workloads and checkpointable service processes feat(sdk): manage checkpointable Compose workloads with native networks Oct 28, 2026
Move Compose preparation, shared ROFS bridge bootstrap and service/process handles into the SDK while preserving the merged snapshot, ownership and command identity contracts. Forward managed runtime configuration through restore and fork without replaying fresh setup.

Keep native IPv4 Compose networking and service-scoped model tunnel entries, and reconnect only owned network rules after restoration. Preserve original process identities and child-first thaw. Add migration regressions, a maintained example, component boundary documentation and helper/distribution CI checks.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
@hky1999
hky1999 force-pushed the hky1999/sdk-managed-compose branch from d932540 to 61606ce Compare October 28, 2026 14:59
@hky1999 hky1999 changed the title feat(sdk): manage checkpointable Compose workloads with native networks feat(sdk): add checkpointable Compose integration on current SDK Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant