feat: introduce UnflushedCheckpointOp::DeleteSnapshot and raise a critical error if snapshots are filtered from tip

[mraszyk]

Snapshot deletions were not recorded as checkpoint operations, so
TipRequest::FilterTipCanisters was the regular mechanism for removing the
directories of deleted snapshots from tip. This change introduces
UnflushedCheckpointOp::DeleteSnapshot, recorded by every code path that can delete a
snapshot:

• CanisterManager::delete_canister_snapshot();
• take_canister_snapshot() and create_snapshot_from_metadata(), for the snapshot
  identified by replace_snapshot;
• the uninstall of a canister that ran out of cycles, in
  SchedulerImpl::charge_canisters_for_resource_allocation_and_usage();
• the deletion of a snapshot's canister, i.e. ReplicatedState::remove_canister() and
  the two subnet split methods (a canister's snapshots live in their own top-level
  snapshots directory, so removing the canister's directory does not cover them).

Renaming a canister needs no equivalent, as rename_canister rejects a canister that
has snapshots (CanisterManagerError::RenameCanisterHasSnapshot).

Like the other operations, these are transient: they are flushed (and taken) before the
checkpoint is written and are never persisted, so neither the checkpoint contents nor
the state hash are affected. flush_unflushed_checkpoint_ops() applies them
sequentially, in the order they were recorded (a snapshot may be taken and then loaded
within the same flush, in which case restore() must see the directory backup() just
created); and the tip thread runs it before persisting page map deltas, so that
backup() copies the files a canister already has in tip and any still-unflushed deltas
are written on top of them.

Every snapshot directory removal from tip is thus an explicit, ordered operation and
FilterTipCanisters is only a safety net for snapshots that disappeared from the state
without a corresponding operation. TipHandler::filter_tip_snapshots() therefore
returns the IDs of the snapshots it removed; if the list is non-empty, the tip thread
logs an error and bumps the new state_manager_tip_snapshots_filtered critical error.
This mirrors state_manager_tip_canisters_filtered.

Deleting a snapshot is hardened so that the operation cannot be forgotten by a new code
path: CanisterSnapshots::remove() and delete_snapshots() now require a
&mut UnflushedCheckpointOps and record the deletion themselves, so there is no way to
remove a snapshot from the state without producing the operation. Callers that only
mutate a single CanisterState and hence have no access to SystemMetadata
(CanisterManager, and the uninstall loop in the scheduler) pass in a temporary
UnflushedCheckpointOps that is merged into the state's operations via the new
UnflushedCheckpointOps::extend(). Accordingly,
CanisterManagerResponse::unflushed_checkpoint_op becomes unflushed_checkpoint_ops,
as replacing a snapshot produces both a DeleteSnapshot and a TakeSnapshot. In the
same spirit, UnflushedCheckpointOps::delete_canister() now takes the CanisterState
rather than just the canister ID and records a DeleteSnapshot for each of the
canister's snapshots, so that they cannot be overlooked.

SnapshotLayout::delete_dir() is tolerant of a missing directory now (as
delete_canister_dir() already was), and its body moved into a free function shared
with the new CheckpointLayout::delete_snapshot_dir(). The tolerance is needed because
a snapshot's directory in tip is only created as a side effect of
CheckpointLayout::snapshot(), and for a snapshot created from uploaded metadata
(which records no TakeSnapshot, as there is nothing to copy from the canister) the
first such call is PageMapType::layout() while handling FlushPageMapDelta: the
snapshot's PageMaps are brand new, hence has_files_in_tip is false, hence
should_flush() holds and they are included in the flush even though they hold no
data. A snapshot created and deleted before any flush sees it in the state therefore
never gets a directory, and flushing its DeleteSnapshot would otherwise fail with
Cannot remove snapshot.: No such file or directory (os error 2), which fatal!s the
tip thread.

Tests: the new filtering_snapshot_from_tip_raises_critical_error covers the critical
error; deleted_snapshot_is_removed_from_tip and
snapshots_of_deleted_canister_are_removed_from_tip cover the flush of the new
operation; snapshot_deletions_record_unflushed_checkpoint_ops and
delete_canister_records_unflushed_checkpoint_ops_for_its_snapshots cover the
operations recorded by execution, as does a new assertion in
snapshot_is_deleted_when_canister_is_out_of_cycles. The idempotency of the new
operation is pinned down by delete_snapshot_dir_is_idempotent (the layout method
directly, including that the enclosing per-canister directory is only removed along
with the canister's last snapshot) and deleting_snapshot_without_tip_directory_is_a_noop
(the flush of the operation end-to-end, for a snapshot built with
CanisterSnapshot::from_metadata()); both fail with the I/O error above if the
NotFound arm is dropped. The online_split expectations in replicated_state.rs drop
two dead canister_snapshots.remove() calls, which removed a snapshot from the wrong
canister's collection and hence were no-ops.

[Copilot]

Pull request overview

Introduces explicit snapshot-deletion checkpoint operations so tip storage remains synchronized with replicated state.

Changes:

• Records and sequentially flushes DeleteSnapshot operations across all deletion paths.
• Makes snapshot-directory deletion idempotent.
• Reports unexpected snapshot filtering as a critical error and adds coverage.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
rs/state_manager/tests/state_manager.rs	Tests deletion flushing and critical errors.
rs/state_manager/src/tip.rs	Flushes snapshot deletions and reports filtering.
rs/state_manager/src/lib.rs	Registers the new critical-error metric.
rs/state_layout/src/state_layout/tests.rs	Tests idempotent directory deletion.
rs/state_layout/src/state_layout.rs	Adds non-creating, idempotent snapshot deletion.
rs/replicated_state/tests/replicated_state.rs	Updates subnet-split expectations.
rs/replicated_state/src/replicated_state.rs	Records snapshot deletions during canister removal.
rs/replicated_state/src/metadata_state.rs	Defines and collects DeleteSnapshot operations.
rs/replicated_state/src/canister_state/canister_snapshots.rs	Records operations when snapshots are removed.
rs/execution_environment/src/scheduler/tests/charging.rs	Verifies out-of-cycles deletion recording.
rs/execution_environment/src/scheduler.rs	Collects deletions during forced uninstall.
rs/execution_environment/src/execution_environment/tests/canister_snapshots.rs	Tests execution-layer operation recording.
rs/execution_environment/src/execution_environment.rs	Merges checkpoint-operation collections.
rs/execution_environment/src/canister_manager/types.rs	Expands responses to multiple operations.
rs/execution_environment/src/canister_manager.rs	Records all manager-driven snapshot deletions.
rs/execution_environment/src/canister_logs.rs	Adapts responses to the new operations field.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to 221b044.

Security Overview

• 🔎 Scanned files: 16 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/f5e9d2bb-29a4-4b64-902a-391d65fd8b2d?scanId=4ac8acb1-773a-4cc6-8585-2a3b134de7a6&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type

Relevant files

Enhancement

► rs/execution_environment/src/canister_logs.rs     Use UnflushedCheckpointOps instead of UnflushedCheckpointOp ► rs/execution_environment/src/canister_manager.rs     Switch to UnflushedCheckpointOps and default usage across message construction ► rs/execution_environment/src/canister_manager/types.rs     Update type to UnflushedCheckpointOps ► rs/execution_environment/src/execution_environment.rs     Aggregate unflushed_checkpoint_ops from responses rather than single op ► rs/execution_environment/src/execution_environment/tests/canister_snapshots.rs     Add tests referencing UnflushedCheckpointOps behavior for snapshots ► rs/execution_environment/src/scheduler.rs     Introduce and merge UnflushedCheckpointOps from deletions ► rs/execution_environment/src/scheduler/tests/charging.rs     Update tests to use UnflushedCheckpointOp in imports ► rs/replicated_state/src/canister_state/canister_snapshots.rs     Modify snapshot removal to record via UnflushedCheckpointOps ► rs/replicated_state/src/metadata_state.rs     Extend UnflushedCheckpointOps with DeleteSnapshot and extend functionality ► rs/replicated_state/src/replicated_state.rs     Record deletion of canister and snapshots as unflushed checkpoint ops ► rs/state_layout/src/state_layout.rs     Add delete_snapshot_directory and path utilities for snapshots ► rs/state_layout/src/state_layout/tests.rs     Add idempotent delete_snapshot_dir test ► rs/state_manager/src/lib.rs     Add tip snapshots filtered critical error constant and metric integration ► rs/state_manager/src/tip.rs     Update TipRequest handling imports to include tip snapshots filter metric ► rs/state_manager/src/tip.rs     Extend TipRequest documentation regarding snapshot deletions and safety net