feat(snapshot): add full checkpoint capture and restore - #1503
Open
appcypher wants to merge 26 commits into
Open
feat(snapshot): add full checkpoint capture and restore#1503appcypher wants to merge 26 commits into
appcypher wants to merge 26 commits into
Conversation
Validate complete checkpoint closures and decode execution and device envelopes before VM construction. Stream verified memory objects once into the inert guest, preload all captured state before first activation, and carry an optional child-owned restore source through the launch contract.\n\nThe public snapshot restore path remains disabled until the workload latch and VM generation activation gate are connected.
Add generation-8 workload freeze and thaw messages and place every agentd-launched process into a cgroup-v2 freezer before exec. Keep normal execution available when the freezer is absent while reporting resumable capture as unavailable. Hold the exact attempt-scoped latch across VM pause, component capture, and root-last publication. Resume and thaw in order, and fail closed when either transition cannot be established.
Persist the captured agent protocol identity in checkpoint resources and restore it through a private console exchange. Resume restored guests only after VM Generation ID processing, release the attempt-scoped workload latch, and synthesize the cached ready handshake. Keep agent and control endpoints unpublished until activation succeeds so clients cannot reach a partially restored runtime.
Add a one-shot network start gate that is selected only for checkpoint restore. Hold smoltcp processing and published-port listener creation until the restored guest has crossed VMGenID processing, workload thaw, and local endpoint publication. Ordinary boots remain immediately active, and the gate is consumed before the poll loop so packet processing has no steady-state check.
Use the existing runtime control endpoint to publish a validated composite checkpoint into a schema-1 snapshot artifact. Materialize immutable checkpoint members without copying runtime-private state and preserve the old artifact until force replacement succeeds.
Materialize installed resumable snapshots into sandbox-owned checkpoint and disk state before construction. Restore captured VM geometry and guest network identity while creating a fresh private qcow2 head for the child.
Capture running checkpoints directly into archives, include exact checkpoint closures when saving installed snapshots, and materialize archives directly into child-owned restore state. Support bounded GNU long-name records for content-addressed closure paths and preserve explicit restore constraints until deferred archive resolution.
Extend explicit snapshot verification to checkpoint-backed artifacts while preserving the released disk verification projection. Stream every referenced memory object, report the verified checkpoint root across the CLI and SDKs, and cover corrupt-memory detection.
Allow checkpoint integrity verification and archive transport across host architectures while keeping construction-time restore admission architecture-specific.
Update the CLI, Rust, Python, TypeScript, and Go snapshot guidance for running checkpoint capture, direct archives, eager child restore, and full checkpoint verification.
Rename the public running-checkpoint mode to full across the CLI and SDKs while retaining structural checkpoint descriptor values and legacy index decoding. Add filesystem-clean capture preparation, disk-only cold restore for installed and direct-archive checkpoints, resume-only full restore command handling, and synchronized user-facing and planning documentation.
Update stale full-capture and GNU long-name assertions, and isolate the sparse archive test from host database state.
Skip ordinary OCI root-disk defaults and growth for installed checkpoint restores so full and disk-only children retain captured disk geometry without mutating hard-linked sealed layers.
Use the representation emitted by the archive writer as the source of truth for encoded size, apparent size, sparse ranges, and transport integrity. This keeps inventories valid when sparse encoding falls back to a dense GNU long-name member and avoids stale pre-scan metadata. Update aggregate limits in constant time with checked replacement, and cover the 101-byte checkpoint qcow2 member path with a direct archive round trip.
Reconstruct runtime-owned passthrough and single-file filesystem state from destination-local paths, including inode identities, open handles, stable directory snapshots, writeback state, and quota accounting.\n\nRestore bootstrap namespaces before device activation, require a ready VM generation driver for full capture, and force resumed workloads onto detached lifecycle ownership before spawning the runtime.
Keep Python missing-snapshot validation synchronous while deferring descriptor parsing and full/disk admission to the shared Rust resolver. Regenerate the Node native bindings so direct archive results are exported, and cover the required native constructor in the contract suite.
Treat an empty round-tripped network interface as defaults rather than an explicit identity override, while still rejecting populated conflicts. Import the Windows access-mode mask required to rebuild captured file handles on native Windows targets.
Make the captured file-handle collection explicit so native Windows can type-check its validation before serializing passthrough state.
Keep the installed checkpoint source separate from a direct archive restore that has already populated child-owned staging. This avoids copying the eager restore closure onto itself and losing the extracted artifact.
Re-evaluate local process ownership after a direct archive descriptor is materialized. Full restores now avoid parent watchdogs and Windows jobs even when their scope was unknown during outer builder dispatch.
|
Comment on lines
+802
to
+810
| if let Some(checkpoint) = response.checkpoint { | ||
| if !response.ok { | ||
| tracing::warn!( | ||
| sandbox = name, | ||
| error = response.error.as_deref().unwrap_or("source resume failed"), | ||
| "checkpoint published but the source runtime did not return to running" | ||
| ); | ||
| } | ||
| return Ok(checkpoint); |
There was a problem hiding this comment.
Quiesced source reported successful
When VM resume or workload thaw fails after checkpoint publication, this branch returns Ok(checkpoint) even though the runtime has transitioned to Quiesced. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.
Knowledge Base Used: Runtime execution orchestration
Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk/rust/lib/sandbox/modify.rs
Line: 802-810
Comment:
**Quiesced source reported successful**
When VM resume or workload thaw fails after checkpoint publication, this branch returns `Ok(checkpoint)` even though the runtime has transitioned to `Quiesced`. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.
**Knowledge Base Used:** [Runtime execution orchestration](https://app.greptile.com/microsandbox/-/custom-context/knowledge-base/superradcompany/microsandbox/-/docs/runtime-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
appcypher
marked this pull request as ready for review
September 1, 2026 18:20
Record monotonic phase timings for stopped and full snapshot creation, archive handling, checkpoint capture, memory restore, and activation. Emit capture telemetry only after the source workload is resumed and thawed so benchmark logging does not extend the measured pause window.
Keep zero detection at 2 MiB while packing non-zero sparse extents into bounded 32 MiB immutable objects. This amortizes hashing, durable object publication, directory work, and restore object reads without changing logical memory coverage.
Do not compare a selected platform manifest body with the parent OCI index digest recorded by multi-platform pulls. Continue binding metadata to the snapshot digest and validating the image config and layer identities.
Persist independent device envelopes and link immutable memory objects with bounded parallel workers after capture. Preserve inventory ordering and existing object verification while reducing time spent in the paused epoch.
Gate the guest heartbeat writer during workload freeze and remove any abandoned temporary heartbeat before acknowledging the checkpoint barrier. Resume writes on thaw or rollback so captured passthrough state never references a transient path absent at restore.
Record the runtime-owned root layout in schema-1 descriptors and preserve flat roots through stopped and full capture, installed and direct archives, disk-only restore, and eager resume. Keep earlier descriptors compatible by defaulting the missing layout to managed. Share checkpoint rollover and restart recovery across managed /dev/vdb uppers and flat /dev/vda roots. Preserve complete raw/qcow2 chains after rollover so later disk snapshots retain post-checkpoint writes. Materialize patched flat roots privately, carry tmpfs state only through full memory snapshots, and retain cloud and user-owned disk-image rejection. Refuse chain-backed growth until the active qcow2 head can be resized without mutating a sealed ancestor.
|
Too many files changed for review (103 files, 100 file limit). Bypass the limit by tagging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Capture a running sandbox as a full snapshot and restore its disk, memory, execution, device, and runtime filesystem state. The same artifact can also be restored disk-only for an ordinary cold boot.
Description
Test Plan
cargo fmt --all -- --checkpassesCloses #250