Repository navigation
feat(adx): retire offline nodes and bound observability overhead - #125
Conversation
f457f5e to
5cf5869
Compare
WenYuLuo
changed the title
WenYuLuo
changed the title
WenYuLuo
changed the title
fix(platform): converge backend cleanup and measure active data traffic Created-by: robbluo Commit-by: robbluo Merged-by: robbluo Description: Paired: [GitHub #138](#138) ↔ [GitCode !110](https://gitcode.com/openJiuwen/agent-dx/merge_requests/110) Timed-out backend Start attempts could retain unknown controllers and repeat reconciliation warnings indefinitely. Keyed backend settlement now waits for the handler and rollback, fences delayed starts, and confirms inventory absence before retiring each failed controller. Cleanup remains per Environment; healthy node admission and unrelated reservations remain available. Pending Start inventory also survives a caller restart through backend discovery and physical-ID cleanup. The same release adds live Ingress and Relay byte counters, including active streams, and reports successful WebSocket 101 separately. Ingress updates per-connection relaxed counters without payload copies or per-chunk registry locks; Relay reuses existing stream counters. Normal command-watch unsubscribe now completes the WebSocket close handshake, avoiding a runtime reset error on successful commands. Pin sandboxd f800327459a104c04a0689940062fc9bb9e2bb9e from inclusionAI/sandboxd#82 and its complete checksum-verified Linux AMD64 bundle. Align the stale AFS readahead contract with master's intentional 16 MiB default. Validation: all 357 adxlet tests and the strengthened 50-case backend protocol suite passed, including blocked cleanup with accepting_allocations=true and an independent reservation, caller restart discovery, and periodic authority retirement. Gateway 90 unit tests, Execd 120 unit tests, three real-socket mock E2E cases, native Rust gates and documentation checks passed. Native Linux amd64 sandboxd formatting, vet, all Go tests and server race tests passed. Release-profile loopback A/B used the same proxy, payload, concurrency and concurrent scrapes for three runs per mode. Plain versus metered medians: 3202/3230 MiB/s, P95 41.08/40.79 us, process CPU 0.77/0.78 s (including Cargo test startup). No material regression was observed in this bounded local probe; this is not a production capacity claim. The native Python SDK also handles a fast PTY command whose peer has already closed before the initial resize: drain buffered terminal frames, preserve the exit code, and still reject missing terminal frames. A real-WebSocket regression reproduces the previous failure; live stress passed 300 fast sessions and three active-close/preserve cases. Forward configured pip sources into E2E Docker builds through a private temporary BuildKit secret; successful builds leave neither the secret nor a config environment variable in the image. Buildkite #202 passed at exact799b450076f19803bb8f80a4b65b05e4aeb2914a: Linux driver452 and native SDK330; AMD64+AFS components, assembly, complete package publication, images and Kubernetes L0+auth. Product/harness revisions match, two physical hosts, missing_checks=[] and cleanup_errors=[]. ARM explicitly excluded. https://buildkite.com/agent-dx/agent-dx/builds/202 Both public archives were fully downloaded and checked by size, SHA256, manifests and AMD64 ELF identity. Live cn-north-4 validation passed ordinary blocked-Start cleanup and sudden caller crash/recovery, including independent healthy work, accepting_allocations=true, empty final backend/controller inventory and no repeated authority warnings. Active 2MiB HTTP transfer counters advanced in both directions before stream close. Normal command watch produced no runtime WebSocket reset error. The complete prewarmed SDK regression passed34/34 with zero skips; initial cold Python OCI preparation exceeded the90-second case budget, retained as a limitation. AKernel integration follow-up after merged#123: inclusionAI/AKernel#125 . <!-- bot4-pr-sync-meta schema_version: 1 source: gitcode gitcode_repo: openJiuwen/agent-dx gitcode_mr: 110 github_repo: openJiuwen-ai/agent-dx github_pr: 138 github_branch: sync/pr-110 --> See merge request: openJiuwen/agent-dx!110
f2c1f87 to
34fcf98
Compare
|
[P1] Align ADX Build 205 with sandboxd/main before merging Reviewed head At Please keep sandboxd pinned to upstream
Local evidence: the identical protobuf/handler probe fails on |
|
@rcyw Thanks for the concrete compatibility check. To clarify the intended contract: cleanup is eventually consistent through periodic reconciliation. A timed-out Start or a lost response may temporarily leave an unowned sandbox; we do not require synchronous proof that no late Start can complete before returning the create failure. Any late sandbox must subsequently be discovered from the backend inventory, compared with the authoritative ownership/runtime identity, and deleted by its physical ID. Once the backend and reconciliation recover and in-flight work settles, repeated reconciliation must converge without leaving residual sandboxes. An empty inventory in one round is not proof that a pending Start cannot appear in a later round. Sandboxd should remain pinned to upstream main ( That said, the implementation issue you identified is valid for the current pair. In Build 205, The ADX compatibility fix and qualification should follow this contract: use the APIs available on sandboxd/main, keep late/orphan discovery active, prevent a failed/retired execution from being adopted as a successful retry, and verify eventual reclamation for Start timeout, lost response, late completion after an initially empty inventory, and caller restart, while preserving valid workloads. Platform/Execd archives and source/protocol pins must then be updated together and tested against that exact backend. The previous |
7309592 to
b8fe1f4
Compare
|
Follow-up after #132: retain the merged runtime baseline and qualify the combined image #132 is now merged into Before merging this PR, please:
This keeps the compatibility fix on upstream sandboxd/main and does not introduce a dependency on sandboxd #82. |
fix(adxlet): reconcile late executions with sandboxd main Created-by: robbluo Commit-by: robbluo Merged-by: robbluo Description: Paired: [GitHub #141](#141) ↔ [GitCode !114](https://gitcode.com/openJiuwen/agent-dx/merge_requests/114) Unknown Start cleanup waits for a keyed-Delete acknowledgement unavailable on sandboxd/main. Use upstream List/Delete and periodic inventory to reclaim late instances after empty scans, failed-controller retirement, or loss of caller state. Failed attempts remain terminal. Inventory is sampled before authority, and Controller queues protect concurrent creates/restores; transient read failures preserve admission. A same-generation owner awaiting its Controller defers cleanup without installing a generation tombstone. Unknown Restore can roll back to Published Paused while retaining the attempted runtime ID. Paused, Failed and Deleted records now stop protecting an execution solely because its ID matches. Both inventory filtering and queued cleanup use the execution state; the Controller rechecks current identity/generation after preceding lifecycle work. Cleanup preserves the checkpoint and Paused state, does not release resources twice, and protects a successful concurrent Resume. Failed Delete is retried by subsequent inventory cycles. Protocol provenance and the verified AMD64 backend archive remain pinned to sandboxd/main `b5dab2958183faa8bcf04955eef7b0ae1c9b10ef`. Validation at `18acb0fe12152628dc0568c2943cc02edccd2c0b`: - 370 adxlet tests passed, 1 native fixture ignored in the regular suite. Five new regressions cover late Restore after Paused rollback, empty startup followed by a late Restore, Delete retries without closing admission, queued cleanup overlapping Resume, and late executions for Failed/Deleted records. These component tests use the real Controller and authority RPC fixture with a fake runtime. - adxlet Clippy, formatting and documentation checks passed. Linux source gates passed, including all-target/all-feature Clippy for the platform and AFS; the earlier macOS workspace attempt hit the Linux-only eventfd dependency. Latest CodeCheck checks are successful. - Dedicated native Sandboxd/runsc: lost Start reply, late Start after an empty inventory, and late Start after reconstructing caller/node state all passed. Every case preserves an unrelated sandbox and the fixture ends with empty inventory. The fixture retries transient inventory sampling timeouts within the same 30-second convergence deadline. These are native Start/adapter cases, not native Restore fault injection. - [Buildkite 208](https://buildkite.com/agent-dx/agent-dx/builds/208) passed all enabled source checks (including all-feature AFS lint), AMD64/AFS packaging, publication, images and Kubernetes L0/auth (2/2, no missing checks or cleanup errors). First attempts hit an AFS RingTimeout and Platform Redis timeouts; each failed step passed one unchanged retry. ARM and PyPI publication were disabled. Follow-up integration: [AKernel #125](inclusionAI/AKernel#125) now consumes the verified matching Build 208 Platform/Execd archives at `b8fe1f44f17c772e71a35ad2dd94cad50dadd6fd`, retaining the Sandboxd mainline pin. All 14 packaged AFS config and 67 Linux build-helper tests passed. [GitHub CI](https://github.com/inclusionAI/AKernel/actions/runs/40707621446) passed all ten required jobs, including source-built runsc Standalone E2E, for this exact AKernel commit. While it ran, AKernel develop merged #132 and advanced Sandboxd to `91d9d05`; #125 now has a gitlink conflict with its retained `b5dab29` pin. These passing results qualify the retained-pin combination; alignment with the new develop runtime bundle is pending. No CN-north-4 upgrade has been performed for this revision. <!-- bot4-pr-sync-meta schema_version: 1 source: gitcode gitcode_repo: openJiuwen/agent-dx gitcode_mr: 114 github_repo: openJiuwen-ai/agent-dx github_pr: 141 github_branch: sync/pr-114 --> See merge request: openJiuwen/agent-dx!114
b8fe1f4 to
ec4f84d
Compare
|
@rcyw Follow-up on the original compatibility finding and the #132 integration review, checked against current head
The completed items are marked above. These are top-level PR comments rather than inline review threads, so there is no Resolve conversation control for them. No CN4 deployment is included in this validation. |
rcyw
left a comment
rcyw
left a comment
There was a problem hiding this comment.
Re-reviewed ec4f84d08c4b8ad4249ce645d5c226b622cb0ce7 on develop 5ded29d, with ADX Build 208 (18acb0fe12152628dc0568c2943cc02edccd2c0b) and sandboxd main 91d9d05edb1b5ea36509a9e2504729daf14cde38. The previous Build 205 fence-compatibility P1 is resolved, and the runtime bundles and runsc-fd-parking checks from #132 are preserved.
Independent local validation passed: 378 adxlet tests; both complete release archives, manifests, file hashes, AMD64 ELF checks and matching Execd payloads; 14 packaged AFS configuration tests; and a source-built runsc image with a normal public-SDK snapshot/restore and data-isolation check. The exact local image was akernel-local/all-in-one:ec4f84d08c4b-amd64-7ba539c1, image ID sha256:724b67970f66b1eaae9aa00cd90c920d8287b60920d779e08c27ca5b0d5889f4.
The native Start part of the previous follow-up is now covered on this final backend: the opt-in fixture passed all three scenarios (lost Start reply, late Start after an empty inventory, and late Start after reconstructing driver/node state). Each preserved the unmanaged sentinel and reclaimed the injected runtime; final cleanup left the backend empty. This used the production driver/controller and an in-process authority RPC fixture against real sandboxd/runsc, with the packaged ADX service stopped to avoid competing reconciliation. It qualifies the native backend, rather than a full deployed control-plane fault E2E.
I recommend fixing the two acceptance-script P2s below and completing native unknown-Restore fault coverage before merging: late Restore after Paused rollback/restart, Delete failure and retry, and cleanup racing a successful Resume. These should verify eventual cleanup while preserving the checkpoint/resource ledger, admission and the successful successor. The existing component regressions use a fake RuntimeDriver, and a normal snapshot/restore does not exercise these faults. This remains a validation gap; I have not reproduced a new server-side defect.
Pin ADX Build 208 archives and source for offline-node retirement, authenticated unregister and Coordinator Watch shutdown. Stop ADX and control Collectors explicitly before Pod termination, and align grace periods with service drain budgets to avoid false worker incidents. Align Sandboxd with develop at upstream main commit 91d9d05edb1b, including its qualified runtime bundles and runsc-fd-parking helper. Reclaim late Start instances through periodic inventory and physical-ID deletion, including after caller restart, without requiring keyed Start fencing. Preserve same-generation authority adoption while its Controller is pending. Collect late Restore executions after rollback to Paused while retaining checkpoints and protecting a concurrent successful Resume. Retain metrics for seven days, trim unused diagnostic series and bound Grafana queries. Match rate windows to scrape cadence and query steps so coarse dashboard views retain counter increments between samples. Add regression coverage for sampling, real Pod deletion, Coordinator restart and exact-once incident accounting. Make lifecycle acceptance honor optional shared-storage paths and Redis ACL accounts. Qualify unknown Restore cleanup against the pinned native backend, including late execution, restart, delete retries and concurrent Resume. Keep deployment instructions and maintenance guidance concise, with detailed evidence in the PR. Signed-off-by: robbluo <luo1442@gmail.com>
ec4f84d to
a98e49f
Compare
|
本轮更新到 未知 Restore 的新后端组合故障回归已补齐，4/4 通过：
测试执行真实 Sandboxd/runsc Pause/Restore/List/Delete，最终物理库存为空，独立 fixture 容器已清理。使用 ADX 验证边界：authority 为进程内 RPC fixture，应用协作/readiness 为测试 hooks；本次证据是原生后端故障回归，不代表完整部署的 Coordinator/API 故障 E2E，也未升级 CN4。新 head 的 GitHub CI 已完成，10 项必需检查全部成功，包括源码镜像构建、SDK 端到端示例、runsc 集成和 Standalone E2E；Docker Hub 发布按 PR 策略跳过。两条 P2 的修复与真实 Redis/存储路径回归也已提交。 |
rcyw
left a comment
rcyw
left a comment
There was a problem hiding this comment.
Re-reviewed a98e49f5d8629862f205e9f374b672af712a6f47. Both previous acceptance-script P2s are fixed. The four lifecycle probe/storage regressions passed locally, including authentication against real Redis 7.4 with a named ACL user and the default user disabled; deployment syntax checks also passed.
I independently ran the new native unknown-Restore fixture: 1 test / all four scenarios passed (late Restore, reconstructed node/driver, Delete failure/retry, and concurrent successful Resume). The checks retained the Paused checkpoint and resource ledger, reused the checkpoint, preserved admission and the healthy peer/successor, and ended with an empty physical backend. This closes the previous native Restore coverage follow-up within the documented production Controller/adapter plus in-process authority/hooks boundary.
The backend reused our source-built image akernel-local/all-in-one:ec4f84d08c4b-amd64-7ba539c1, image ID sha256:724b67970f66b1eaae9aa00cd90c920d8287b60920d779e08c27ca5b0d5889f4. Its ADX 18acb0fe, sandboxd 91d9d05 and complete pinned runsc bundle are unchanged by this revision.
There is one additional P2 in the new runner's handling of an external Cargo target directory, detailed inline. After the successful native run, repeating the runner with --compile-only and the same cache failed in 0.8 seconds before executing tests. Please isolate the temporary tree's build artifacts or retain a stable source path so the qualification entry can be rerun reliably.
rcyw
left a comment
rcyw
left a comment
There was a problem hiding this comment.
Approved at a98e49f5d8629862f205e9f374b672af712a6f47.
Both previous acceptance-script findings are fixed. The four lifecycle probe/storage regressions and deployment syntax checks passed locally, and independent native backend qualification passed all four unknown-Restore scenarios, closing the earlier coverage follow-up within the documented fixture boundary.
The remaining external Cargo cache issue in the new test runner is non-blocking and can be fixed in a follow-up. It produces a visible build failure before tests execute; the completed native backend validation remains valid. I recommend isolating the temporary source tree's build artifacts to make repeated cached runs reliable.
Update ADX Platform/Execd archives and the source gitlink to Build 208, commit
18acb0fe12152628dc0568c2943cc02edccd2c0b(ADX #141). Rebase on develop5ded29d5809f7f279c7ae122b3312d6912207fc6and use its Sandboxd main pin91d9d05edb1b5ea36509a9e2504729daf14cde38, including the qualified gVisor/Firecracker runtime bundles andrunsc-fd-parkingpayload checks from #132.Unknown Start cleanup uses ordinary List/Delete by physical runtime ID. Periodic inventory reclaims instances that appear after an empty scan, failed-controller retirement, or loss of caller/node state. Controller queues protect concurrent creates/restores; a same-generation authority record awaiting its Controller defers cleanup without installing a generation tombstone. Transient inventory/authority read failures preserve admission.
Unknown Restore may roll back to Published Paused while retaining the attempted runtime ID. Paused, Failed and Deleted records now stop protecting a late execution solely because its ID matches. Queued cleanup rechecks the latest Controller state, generation and runtime ID, retains the Paused checkpoint and resource ledger, protects successful concurrent Resume, and retries failed deletion on subsequent scans. Both archives have verified SHA256, complete package hashes, clean AMD64/AFS manifests and matching Execd payloads.
Current validation:
b5dab29with runsc qualification: lost Start reply, late Start after an empty inventory, and late Start after reconstructing caller/node state all passed. Each preserves a live peer and eventually removes the injected instance; final cleanup asserts an empty backend. Transient inventory sampling timeouts are retried within the same 30-second convergence deadline.18acb0f(Build 208), source-built Sandboxd91d9d05and its complete gVisorrelease-20261012.0-akernel.1bundle: late execution after Published Paused rollback; late execution after reconstructing Adxlet/driver with an empty initial inventory; three Delete failures followed by retry; and cleanup queued behind a successful concurrent Resume. The test preserves/reuses the real checkpoint, checks the ledger and admission, retains a running peer/successor, and asserts an empty physical inventory after teardown. The maintained runner isdeploy/tests/native_restore/run.py(Python 3.12+). This is native-backend qualification with the production Controller/adapter and an in-process authority, not deployed Coordinator/API fault E2E.--shared-storage-pathand supports decoded Redis ACL username/password credentials. Three storage cases plus real isolated-Redis authentication modes pass; the probe contracts run in CI. Deployment syntax checks pass.runsc-fd-parkinghelper.ec4f84dpassed all ten required jobs for Sandboxd91d9d05with ADX Build 208, including source-built image/config/systemd/gVisor validation, SDK end-to-end examples, runsc SDK integration and Standalone E2E. Docker Hub publication is skipped for pull requests. This revision has not been deployed to CN-north-4; the cluster results below qualify the previous image.Latest head
a98e49f: native Restore and lifecycle probe regressions passed. GitHub CI passed all ten required jobs, including the new real-Redis probe contracts, source-built image, SDK end-to-end examples, runsc integration and Standalone E2E; Docker Hub publication was skipped.Integration status: the Sandboxd gitlink conflict with #132 is resolved by adopting develop's
91d9d05pin. The PR remains one commit on the latest verified develop baseline.Fix Pod termination and false worker incidents. The all-in-one image's systemd stop signal does not stop adxctl or Collector PID 1. Explicit preStop hooks stop ADX and send SIGTERM to control Collectors; Pod grace covers the service budgets (node 150s, Coordinator 120s, Ingress/API 210s). Closing Watch streams prevents the old Coordinator from exhausting its stop timeout, restarting under its supervisor and fencing the replacement. A pre-fix Coordinator replacement with four unchanged workers increased the authoritative Redis incident count by four; the qualified replacement now adds zero.
Keep observability bounded: default Prometheus retention is seven days; business sampling stays 15 seconds and self monitoring uses 30 seconds. Grafana's datasource now matches the 15-second cadence, Collector rate queries specify 30 seconds, and history queries use step-aware rate windows. The 200-point target, one-minute minimum interval/overview refresh, instant tables/totals, scoped Environment detail, aggregate Relay history, 200 log lines and 20 create traces remain. Grafana rounds query steps, so the target is approximate. Remove unused request-QPS recording and Collector diagnostic histogram families while preserving application/OTLP metrics and queue/refused/send-failed health metrics.
deploy/observability.mdremains deleted.Earlier dashboard/shutdown validation:
Previous CN-north-4 deployment (
akernel, core revision 112; monitor revision 44, ADX Build 205 / Sandboxdf800327):pr125-456ab90-adx205-afs, digestsha256:e2c984a93810056fadd780621af19fcfa55a532f1dfba03a55ae4ad6a6b96534. Image inputs are unchanged by the final Helm/acceptance updates. Six control/node Pods match the ADX manifest; AFS producer Pod identities are preserved and all four consumer mounts are healthy.Earlier GitHub CI passed for commit
fcb51ec4eacb6be28a195697ee015186d141df4a: SDK unit tests on Python 3.10–3.14, lint/types, distributions, deployment syntax, ADX release/deployment contracts and Standalone E2E. Docker Hub publication is intentionally skipped for pull requests.Dashboard screenshots
Captured from CN-north-4, namespace
akernel, withcluster=akernel-cceandadx_env=akernel-cceat 2026-10-28 17:03 UTC+8. Both views use the last six hours. The cluster is idle at capture time; the history includes regression and fault-injection activity.Scheduling overview: reachable/schedulable workers, completed-create throughput and create outcomes.
Data-plane overview: response counts by HTTP class and Ingress/Relay traffic over the selected window. Counts use Prometheus counter extrapolation and can be fractional.