Skip to content

feat(snapshot): add full checkpoint capture and restore - #1503

Open
appcypher wants to merge 26 commits into
appcypher/composite-checkpoint-prfrom
appcypher/checkpoint-restore-clone
Open

feat(snapshot): add full checkpoint capture and restore#1503
appcypher wants to merge 26 commits into
appcypher/composite-checkpoint-prfrom
appcypher/checkpoint-restore-clone

Conversation

@appcypher

@appcypher appcypher commented Sep 1, 2026

Copy link
Copy Markdown
Member

TL;DR

Capture a running sandbox as a full snapshot and restore its disk, memory, execution, device, and runtime filesystem state. The same artifact can also be restored disk-only for an ordinary cold boot.

Description

  • Add full snapshot capture for running sandboxes, with guest workload quiescing, same-epoch runtime checkpoint publication, and explicit source resume.
  • Restore full snapshots eagerly into child-owned disk and runtime state, gate workload and network activation on freshness, and keep the source artifact independent after construction.
  • Stream full snapshots directly to archives without installing an intermediate snapshot directory or index row, and verify the complete checkpoint closure before restore.
  • Add disk-only restore for installed and archived full snapshots, producing a fresh writable qcow2 head while discarding captured memory and execution state.
  • Expose the behavior consistently through the CLI and Rust, Python, TypeScript, and Go SDKs, including warnings when a command is ignored during full restore.
  • Capture and restore passthrough filesystem handles and directory iteration state on Unix and Windows, and report unsupported checkpoint capabilities explicitly.
msb snapshot create worker-checkpoint --from worker --full
msb run --name worker-copy --from-snapshot worker-checkpoint
msb run --name worker-cold --from-snapshot worker-checkpoint --disk-only
let snapshot = Snapshot::builder("worker-checkpoint")
    .from_sandbox("worker")
    .full()
    .create()
    .await?;

let sandbox = Sandbox::builder("worker-cold")
    .from_snapshot("worker-checkpoint")
    .disk_only()
    .create()
    .await?;
snapshot = await Snapshot.create(
    "worker-checkpoint",
    from_sandbox="worker",
    full=True,
)

sandbox = await Sandbox.create(
    "worker-cold",
    from_snapshot="worker-checkpoint",
    disk_only=True,
)
const snapshot = await Snapshot.builder("worker-checkpoint")
  .fromSandbox("worker")
  .full()
  .create();

const sandbox = await Sandbox.builder("worker-cold")
  .fromSnapshot("worker-checkpoint")
  .diskOnly()
  .create();
snapshot, err := m.Snapshot.Create(ctx, m.SnapshotCreateOptions{
    Name:        "worker-checkpoint",
    FromSandbox: "worker",
    Full:        true,
})

sandbox, err := m.CreateSandbox(ctx, "worker-cold",
    m.WithFromSnapshot("worker-checkpoint"),
    m.WithSnapshotDiskOnly(),
)

Test Plan

  • cargo fmt --all -- --check passes
  • Filesystem, runtime, and Rust SDK test suites pass against the stacked libkrun checkpoint substrate
  • Python, TypeScript, and Go SDK build, lint, type, unit, and live smoke checks pass
  • Full, direct-archive, and disk-only variants pass live on macOS AArch64 with HVF
  • Full, direct-archive, and disk-only variants pass live on Linux x86-64 with KVM
  • Full, direct-archive, and disk-only variants pass live on Windows AArch64 with WHP (Surface unavailable; final rerun pending)

Closes #250

Validate complete checkpoint closures and decode execution and device envelopes before VM construction. Stream verified memory objects once into the inert guest, preload all captured state before first activation, and carry an optional child-owned restore source through the launch contract.\n\nThe public snapshot restore path remains disabled until the workload latch and VM generation activation gate are connected.
Add generation-8 workload freeze and thaw messages and place every agentd-launched process into a cgroup-v2 freezer before exec. Keep normal execution available when the freezer is absent while reporting resumable capture as unavailable.

Hold the exact attempt-scoped latch across VM pause, component capture, and root-last publication. Resume and thaw in order, and fail closed when either transition cannot be established.
Persist the captured agent protocol identity in checkpoint resources and restore it through a private console exchange. Resume restored guests only after VM Generation ID processing, release the attempt-scoped workload latch, and synthesize the cached ready handshake.

Keep agent and control endpoints unpublished until activation succeeds so clients cannot reach a partially restored runtime.
Add a one-shot network start gate that is selected only for checkpoint restore. Hold smoltcp processing and published-port listener creation until the restored guest has crossed VMGenID processing, workload thaw, and local endpoint publication.

Ordinary boots remain immediately active, and the gate is consumed before the poll loop so packet processing has no steady-state check.
Use the existing runtime control endpoint to publish a validated composite checkpoint into a schema-1 snapshot artifact. Materialize immutable checkpoint members without copying runtime-private state and preserve the old artifact until force replacement succeeds.
Materialize installed resumable snapshots into sandbox-owned checkpoint and disk state before construction. Restore captured VM geometry and guest network identity while creating a fresh private qcow2 head for the child.
Capture running checkpoints directly into archives, include exact checkpoint closures when saving installed snapshots, and materialize archives directly into child-owned restore state. Support bounded GNU long-name records for content-addressed closure paths and preserve explicit restore constraints until deferred archive resolution.
Extend explicit snapshot verification to checkpoint-backed artifacts while preserving the released disk verification projection. Stream every referenced memory object, report the verified checkpoint root across the CLI and SDKs, and cover corrupt-memory detection.
Allow checkpoint integrity verification and archive transport across host architectures while keeping construction-time restore admission architecture-specific.
Update the CLI, Rust, Python, TypeScript, and Go snapshot guidance for running checkpoint capture, direct archives, eager child restore, and full checkpoint verification.
Rename the public running-checkpoint mode to full across the CLI and SDKs while retaining structural checkpoint descriptor values and legacy index decoding.

Add filesystem-clean capture preparation, disk-only cold restore for installed and direct-archive checkpoints, resume-only full restore command handling, and synchronized user-facing and planning documentation.
Update stale full-capture and GNU long-name assertions, and isolate the sparse archive test from host database state.
Skip ordinary OCI root-disk defaults and growth for installed checkpoint restores so full and disk-only children retain captured disk geometry without mutating hard-linked sealed layers.
Use the representation emitted by the archive writer as the source of truth for encoded size, apparent size, sparse ranges, and transport integrity. This keeps inventories valid when sparse encoding falls back to a dense GNU long-name member and avoids stale pre-scan metadata.

Update aggregate limits in constant time with checked replacement, and cover the 101-byte checkpoint qcow2 member path with a direct archive round trip.
Reconstruct runtime-owned passthrough and single-file filesystem state from destination-local paths, including inode identities, open handles, stable directory snapshots, writeback state, and quota accounting.\n\nRestore bootstrap namespaces before device activation, require a ready VM generation driver for full capture, and force resumed workloads onto detached lifecycle ownership before spawning the runtime.
Keep Python missing-snapshot validation synchronous while deferring descriptor parsing and full/disk admission to the shared Rust resolver.

Regenerate the Node native bindings so direct archive results are exported, and cover the required native constructor in the contract suite.
Treat an empty round-tripped network interface as defaults rather than an
explicit identity override, while still rejecting populated conflicts.

Import the Windows access-mode mask required to rebuild captured file
handles on native Windows targets.
Make the captured file-handle collection explicit so native Windows can
type-check its validation before serializing passthrough state.
Keep the installed checkpoint source separate from a direct archive restore
that has already populated child-owned staging. This avoids copying the
eager restore closure onto itself and losing the extracted artifact.
Re-evaluate local process ownership after a direct archive descriptor is
materialized. Full restores now avoid parent watchdogs and Windows jobs even
when their scope was unknown during outer builder dispatch.
@greptile-apps

greptile-apps Bot commented Sep 1, 2026

Copy link
Copy Markdown

Sequence Diagram

sequenceDiagram
  participant C as CLI / SDK
  participant S as Source sandbox
  participant A as Guest agent
  participant R as Runtime
  participant V as VM
  participant P as Snapshot publisher
  participant D as Restored sandbox
  C->>R: Create full snapshot
  R->>A: Freeze workload
  A-->>R: Workload frozen
  R->>V: Pause VM and capture state
  R->>P: Publish checkpoint closure root-last
  R->>V: Resume source VM
  R->>A: Thaw workload
  P-->>C: Full snapshot artifact
  C->>D: Restore full or disk-only
  D->>P: Verify descriptor and closure
  alt Full restore
    D->>D: Materialize disk, memory, device, and runtime state
    D->>V: Resume captured execution
  else Disk-only restore
    D->>D: Copy sealed disk layers and create writable qcow2 head
    D->>V: Cold boot
  end
Loading

Fix all with Greploop Fix All in Codex Fix All in Claude Code

Prompt To Fix All With AI
### Issue 1
sdk/rust/lib/sandbox/modify.rs:802-810
**Quiesced source reported successful**

When VM resume or workload thaw fails after checkpoint publication, this branch returns `Ok(checkpoint)` even though the runtime has transitioned to `Quiesced`. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "fix(snapshot): detach deferred full rest..." | Re-trigger Greptile

Comment on lines +802 to +810
if let Some(checkpoint) = response.checkpoint {
if !response.ok {
tracing::warn!(
sandbox = name,
error = response.error.as_deref().unwrap_or("source resume failed"),
"checkpoint published but the source runtime did not return to running"
);
}
return Ok(checkpoint);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Quiesced source reported successful

When VM resume or workload thaw fails after checkpoint publication, this branch returns Ok(checkpoint) even though the runtime has transitioned to Quiesced. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.

Knowledge Base Used: Runtime execution orchestration

Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk/rust/lib/sandbox/modify.rs
Line: 802-810

Comment:
**Quiesced source reported successful**

When VM resume or workload thaw fails after checkpoint publication, this branch returns `Ok(checkpoint)` even though the runtime has transitioned to `Quiesced`. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.

**Knowledge Base Used:** [Runtime execution orchestration](https://app.greptile.com/microsandbox/-/custom-context/knowledge-base/superradcompany/microsandbox/-/docs/runtime-execution.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

@appcypher
appcypher marked this pull request as ready for review September 1, 2026 18:20
Record monotonic phase timings for stopped and full snapshot creation, archive handling, checkpoint capture, memory restore, and activation. Emit capture telemetry only after the source workload is resumed and thawed so benchmark logging does not extend the measured pause window.
Keep zero detection at 2 MiB while packing non-zero sparse extents into bounded 32 MiB immutable objects. This amortizes hashing, durable object publication, directory work, and restore object reads without changing logical memory coverage.
Do not compare a selected platform manifest body with the parent OCI index digest recorded by multi-platform pulls. Continue binding metadata to the snapshot digest and validating the image config and layer identities.
Persist independent device envelopes and link immutable memory objects with bounded parallel workers after capture. Preserve inventory ordering and existing object verification while reducing time spent in the paused epoch.
Gate the guest heartbeat writer during workload freeze and remove any abandoned temporary heartbeat before acknowledging the checkpoint barrier. Resume writes on thaw or rollback so captured passthrough state never references a transient path absent at restore.
Record the runtime-owned root layout in schema-1 descriptors and preserve flat roots through stopped and full capture, installed and direct archives, disk-only restore, and eager resume. Keep earlier descriptors compatible by defaulting the missing layout to managed.

Share checkpoint rollover and restart recovery across managed /dev/vdb uppers and flat /dev/vda roots. Preserve complete raw/qcow2 chains after rollover so later disk snapshots retain post-checkpoint writes.

Materialize patched flat roots privately, carry tmpfs state only through full memory snapshots, and retain cloud and user-owned disk-image rejection. Refuse chain-backed growth until the active qcow2 head can be resized without mutating a sealed ancestor.
@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown

Too many files changed for review (103 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant