Skip to content

vm: lazy memory restore via UFFD; stop systemd unit on restore failure#53

Merged
meAmitPatil merged 9 commits into
mainfrom
amit/uffd-lazy-restore
May 13, 2026
Merged

vm: lazy memory restore via UFFD; stop systemd unit on restore failure#53
meAmitPatil merged 9 commits into
mainfrom
amit/uffd-lazy-restore

Conversation

@meAmitPatil
Copy link
Copy Markdown
Contributor

@meAmitPatil meAmitPatil commented May 12, 2026

Summary

Switches sandbox restores from Firecracker's synchronous File memory backend (CRC64-verified up front) to a userfaultfd handler that serves pages on demand. Eliminates the ~22% host CPU previously burnt on CRC64 and unblocks concurrent-create throughput.

What changed

  • New internal/vm/uffd package — pure-Go UFFD handler, one per VM. Receives the userfaultfd via SCM_RIGHTS, serves UFFDIO_COPY on demand, runs a background prefetcher.
  • Access-pattern recording at template build time — the build pipeline restores the just-snapshotted VM under a recording UFFD handler and writes a page-access order (access.log) into the template dir. Settle detector uses fault-rate convergence (not a fixed timeout), so each template self-tunes.
  • Ordered prefetch on sandbox restore — when access.log exists, the handler replays it in the background; on cold templates we fall back to sequential.
  • RestoreSnapshotUffdWithOverrides — File-backend variant kept intact for inPlace resume + the UffdEnabled=false circuit breaker.

Safety

  • VMD_UFFD_ENABLED=false flips fresh restores back to File backend, no redeploy needed. Existing UFFD-served VMs keep running on their handlers.
  • VMD_UFFD_PREFETCH_ENABLED=false keeps UFFD but disables ordered prefetch and the build-time recording.
  • Tested fixes from independent review:
    • record-build-* VMs use the build- prefix so the reconciler/persistence layer treats them as ephemeral (avoids BoltDB orphans on vmd crash).
    • Handshake outcome ack'd back to the manager — no silent "Firecracker LoadSnapshot returned success but handler died" zombies.
    • AcceptUnix watchdog wakes on ctx cancel.
    • UFFD LoadSnapshot bounded by 30s timeout.

Open in Stage

@stage-review
Copy link
Copy Markdown

stage-review Bot commented May 12, 2026

Ready to review this PR? Stage has broken it down into 6 individual chapters for you:

Title
1 Implement core UFFD handler and kernel ABI
2 Add prefetcher and access pattern recorder
3 Extend Firecracker client for UFFD backend
4 Integrate UFFD into the VM Manager
5 Wire UFFD recording into template builds
6 Update CLI and reconciler for UFFD safety
Open in Stage

Chapters generated by Stage for commit c919a35 on May 13, 2026 2:52am UTC.

@meAmitPatil meAmitPatil force-pushed the amit/uffd-lazy-restore branch from dfe5d6e to a7ffc11 Compare May 12, 2026 21:23
@meAmitPatil meAmitPatil marked this pull request as ready for review May 12, 2026 22:20
@chatgpt-codex-connector
Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

Comment thread internal/vm/uffd/handler.go Outdated
Comment thread internal/vm/uffd/handler.go
Comment thread internal/vm/uffd/ioctl.go Outdated
@meAmitPatil meAmitPatil merged commit 3c258c2 into main May 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants