feat(cayenne): default-on adaptive cold layout from observed filters (F4) by lukekim · Pull Request #11973 · spiceai/spiceai

lukekim · 2026-07-21T23:57:17Z

Summary

Default-on adaptive cold layout (F4): when sort_columns / cold clustering are empty (production default), Cayenne records which columns appear in scan filters and uses the hottest ones to sort compaction / sort-and-rewrite output.
Zone maps on the rewritten files prune selective queries without spicepod maintained_aggregates or cayenne_sort_columns.
Explicit operator sort_columns always wins; cold-tier Z-order resolution prefers observations over PK fallback when clustering is unset.
Output-ordering attestation remains limited to configured sort columns (auto layout can change over time).

How it works

FilterColumnObservations (bounded histogram, max 32 columns) on each table.
TableProvider::scan records distinct filter column names (nested AND/OR supported).
effective_sort_columns_for_rewrite() → configured sort columns, else top-2 observed columns in schema.
Compaction rewrite and sort_and_rewrite_data sort by that list when non-empty.

Validation

Check	Result
`cargo test -p cayenne --lib predicate_stats`	4 passed
`cargo test -p cayenne --test adaptive_layout_test`	3 passed
`cargo clippy -p cayenne --no-deps --lib --tests --bench layout_amp_default_on -- -D warnings`	clean
`cargo bench -p cayenne --bench layout_amp_default_on`	see below

Bench anchor (100k rows, 64 files, ~5% selective)

random_append (default empty sort_columns): amp=20.0×
sorted_by_filter_col (auto-cluster target):  amp=1.2×
ratio: 16.0× fewer bytes after adaptive layout
filter_column_extract/extract_3_filters: ~88 ns

Test plan

Unit tests for histogram ranking / schema filter / nested exprs
Integration: learn columns from scans; configured override; sort-and-rewrite preserves rows
Layout amp bench asserts ≥5× amp reduction
CI full suite on PR
Optional: run selective filter + compaction under default spicepod on a CDC table and confirm files_scanned drops post-compact

…(F4) When sort_columns is empty (the production default), record filter columns on every scan and sort compaction / sort-and-rewrite by the hottest observed columns so zone maps prune selective queries without spicepod setup. Explicit cayenne_sort_columns still wins; cold-tier Z-order falls back through observations before the primary key. Includes unit tests, integration tests, and a layout read-amp bench (16× fewer bytes vs append layout at 5% selectivity).

github-actions · 2026-07-21T23:57:28Z

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

area/docs

Passing checks:

✅ Title meets minimum length requirement (10 characters)
✅ No banned labels detected
✅ Has a label from required category kind/
✅ Has a label from required category area/
✅ Has at least one assignee: lukekim

Copilot

Pull request overview

Implements Cayenne “F4” default-on adaptive cold-layout selection: when sort_columns is unset/empty, Cayenne observes which columns appear in scan pushdown filters and uses the hottest observed columns to drive rewrite/compaction sorting (improving zone-map pruning without requiring spicepod config).

Changes:

Add per-table filter-column observation histogram and record filter columns on TableProvider::scan.
Use “effective” rewrite sort columns (configured sort_columns first, else top observed columns) for sort-and-rewrite and compaction rewrite paths, and prefer observations over PK fallback for cold-tier clustering when unset.
Add integration tests + a criterion bench to validate/anchor the default-on adaptive layout behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
crates/cayenne/tests/adaptive_layout_test.rs	New integration tests covering learning from scans, configured override, and rewrite correctness.
crates/cayenne/src/provider/table.rs	Plumbs shared observations into the provider; uses effective sort columns for compaction/rewrite; records filters during scans.
crates/cayenne/src/provider/predicate_stats.rs	New bounded histogram implementation + unit tests for column extraction/ranking.
crates/cayenne/src/provider/mod.rs	Wires in the new `predicate_stats` module.
crates/cayenne/Cargo.toml	Registers new `layout_amp_default_on` benchmark target.
crates/cayenne/benches/layout_amp_default_on.rs	Adds read-amplification anchor bench and filter-column extraction microbench.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

crates/cayenne/src/provider/table.rs:11391

The doc comments above sort_stream_by_columns look like they still include a leftover # Errors sentence from the old sort_stream implementation, and now duplicate / slightly misdescribe the current behavior. Please simplify to a single description + single # Errors section so the docs stay accurate.

- predicate_stats: deterministic eviction victim (break ties by column name) so the observed set — and therefore auto-layout — no longer depends on randomized HashMap iteration order under column churn - table.rs: fix the duplicated/misleading `sort_stream_by_columns` `# Errors` doc; missing/invalid sort columns are skipped with a warning and the stream returned unsorted, not surfaced as an error - docs/cayenne: document the cold-tier clustering-key precedence (clustering cols -> sort cols -> hot observed filters -> PK), update the cheat-sheet default, and add a Document changelog row - align new adaptive_layout_test / layout_amp_default_on license headers to the 2024-2026 repo standard - rustfmt this PR's new code

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

        use datafusion_execution::TaskContext;

+        if sort_columns.is_empty() {
+            return Ok(stream);
+        }
+
        // Create a task context with default memory pool and runtime settings
        // This will use the configured spill directory and compression settings
        let task_ctx = Arc::new(TaskContext::default());

        tracing::debug!(


Copilot AI review requested due to automatic review settings July 21, 2026 23:57

lukekim requested a review from a team as a code owner July 21, 2026 23:57

github-actions Bot added area/config area/tests kind/feature size/xl labels Jul 21, 2026

Copilot started reviewing on behalf of lukekim July 21, 2026 23:57 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread crates/cayenne/src/provider/table.rs Outdated

Comment thread crates/cayenne/src/provider/predicate_stats.rs

phillipleblanc previously approved these changes Jul 22, 2026

View reviewed changes

Comment thread crates/cayenne/src/provider/predicate_stats.rs

lukekim self-assigned this Jul 22, 2026

lukekim added this to the v2.2.0 milestone Jul 22, 2026

Merge branch 'trunk' into lukim/cayenne-adaptive-cold-layout

86cc360

Copilot AI review requested due to automatic review settings July 22, 2026 05:06

Copilot started reviewing on behalf of lukekim July 22, 2026 05:06 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread crates/cayenne/src/provider/table.rs

Comment thread crates/cayenne/tests/adaptive_layout_test.rs Outdated

Comment thread crates/cayenne/benches/layout_amp_default_on.rs Outdated

Copilot AI review requested due to automatic review settings July 22, 2026 05:30

lukekim dismissed phillipleblanc’s stale review via 4fe1a60 July 22, 2026 05:30

github-actions Bot added the area/docs label Jul 22, 2026

Copilot started reviewing on behalf of lukekim July 22, 2026 05:31 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cayenne): default-on adaptive cold layout from observed filters (F4)#11973

feat(cayenne): default-on adaptive cold layout from observed filters (F4)#11973
lukekim wants to merge 3 commits into
trunkfrom
lukim/cayenne-adaptive-cold-layout

lukekim commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lukekim commented Jul 21, 2026

Summary

How it works

Validation

Bench anchor (100k rows, 64 files, ~5% selective)

Test plan

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

Passing checks:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 21, 2026 •

edited

Loading