Plan-time metadata-only DELETE: drop fully-covered data files from manifest bounds without scanning

[paultmathew]

Summary

Add a plan-time optimization for DELETE where, if a DELETE's predicate provably covers all rows of a data file (from the file's partition and/or manifest column bounds), the file is dropped as a metadata-only delete — marked deleted in a new manifest — without scanning it and without writing a positional/equality delete file. Files only partially covered fall back to the existing scan + row-level delete path.

Motivation / why a new approach

I took a first cut at metadata-only deletes in #1178, but that PR resolves whole-file deletes at flush time — i.e., after the rows have already been scanned, once it observes that every live row of a file was deleted. In practice that path ended up entangled with manifest-rewrite / snapshot-summary / compaction-adjacent concerns, and the PR grew unwieldy without delivering the win that actually matters for large partitioned tables: not reading the data at all.

I'd like to step back and pursue the optimization at plan time instead, which is both more self-contained and where the real I/O savings are (e.g. DELETE ... WHERE dt = '2026-01-01' on a table partitioned by dt should touch zero data files).

Proposed approach

At DELETE planning, for each candidate data file:

1. Evaluate the delete predicate against the file's partition values and manifest column bounds (lower/upper bounds, null counts).
2. If the predicate provably selects every row of the file → drop the file metadata-only (new manifest entry marked DELETED), skip the scan.
3. If the predicate provably selects no rows → prune the file entirely (no-op).
4. Otherwise (partial / undecidable) → fall back to the current scan + row-level delete.

Relationship to #341

This is essentially the write-path counterpart to #341 ([Predicate Pushdown] Filtering deletes based on row counts). #341 asks "can this scan predicate match no rows of a delete file, so we can ignore it?"; this asks "does this delete predicate match all rows of a data file, so we can drop it whole?" Both need the same bounds-vs-predicate evaluator over manifest metrics — it would be worth sharing that machinery.

Scope / non-goals

• In scope: the plan-time whole-file drop + fallback, using existing manifest partition/bounds metadata.
• Non-goals: compaction / manifest rewriting, snapshot-summary accounting beyond what a normal delete already does, and equality-delete-file pruning on reads (that is [Predicate Pushdown] Filtering deletes based on row counts #341).

Open questions

• Reuse vs. share the evaluator with [Predicate Pushdown] Filtering deletes based on row counts #341 — is there appetite to factor out a common bounds/predicate util?
• How strict to be about bound tightness (Iceberg bounds can be truncated) — only drop on exact/closed-interval containment.

[Tmonster]

This has been on TODO list! Would love to review a PR!

I think a good start is to implement this for partitions only.

In the planning stage make sure you remove the IcebergScan operator from lower in the plan. It might be that in IcebergDelete::PlanDelete you will need to remove the IcebergScan from the children, and modify IcebergDelete to have different paths based on if it can be a MetadataOnly delete.

If you do tackle this, I would love to see some regression tests in Regression.yml 😄

[Tishj]

My quick thoughts on an implementation for this:
I think we'll want it to be more granular than a "MetadataOnly" delete. That would only be possible if all data files hit by the delete filter are entirely covered.

I think instead we'll want to integrate this into the IcebergMultiFileList, being able to make the scan aware of the delete predicate, so it can treat this like an extra layer of filters, and registering the data_file.file_path of the file being skipped.
Then the delete operator can use this list to add to altered_manifests, causing the manifest(s) containing the data file(s) to get rewritten.

[djouallah]

"That would only be possible if all data files hit by the delete filter are entirely covered." yes but that's a very common use case with truncate table and create or replace tables

[Tishj]

I'm only suggesting an improvement, this shouldn't take away from the benefit of the optimization, just make it applicable to more general use cases

[paultmathew]

Thanks for the feedback — I'm going to split this into two PRs to keep them reviewable.

PR1 — predicate-aware IcebergScan, monotonic partition keys only.

The flow is:

1. Strict evaluator. Add an "all rows match" evaluator in iceberg_predicate.cpp — the strict-projection mirror of the existing inclusive MatchBounds. Given a file's stats + partition transform, it answers "does the predicate hold for every row?" (< uses the upper bound, > the lower, = requires lower == upper, AND requires all conjuncts; anything it can't prove → false).
2. Make IcebergMultiFileList predicate-aware. Pass the delete predicate down to the scan's file list. During enumeration, each file is evaluated against its partition value with the strict evaluator — an extra layer on top of the existing inclusive pruning.
3. Register fully-covered files. If every row of a file is covered, the scan skips it (doesn't emit its rows) and records its data_file.file_path in a side-list. Partially-covered / unprovable files fall through to the normal scan + row-level delete.
4. Drop via altered_manifests. The delete operator adds those registered paths to altered_manifests, reusing the Metadata-only delete when a DELETE removes every live row of a data file #1178 commit path: the containing manifest(s) are rewritten with those entries marked DELETED.
5. Stats update for free. Because the rewrite goes through CreateFromEntries, the manifest counts (deleted_files_count / deleted_rows_count / existing_), partition field summaries, and the derived snapshot summary (deleted-data-files, deleted-records, total-) are all recomputed from the entry statuses — deleted_rows_count += record_count, which is exactly right since these files are fully deleted.

Scope for PR1:

1. Partition keys only, and only monotonic transforms (identity, year/month/day/hour, truncate) — already handled in MatchBounds. Boundary/straddling partitions and non-monotonic transforms (bucket) conservatively fall back to the scan.
2. Because it's per-file, the all-covered case drops out for free: if every file the delete touches is covered (e.g. an unfiltered DELETE), the scan reads nothing at all — which also serves the whole-table use case @djouallah raised.

PR2 — extend to non-partition data columns.

Same strict evaluator, but fed each file's column lower/upper bounds instead of the partition value. This needs truncation-direction guards (a truncated upper bound can't prove = V or < Y) — the same care #341 needs on the read side, so I'll aim to share that evaluator.

Regression tests in Regression.yml as suggested.

Please let me know if this looks okay.

[Tishj]

Yea that sounds good to me 👍
In V4 we'll have tight_bounds that should indicate whether the bounds are truncated or not, but without that, we have to be conservative.

When true, lower_bound and upper_bound must be equal to the min and max values