Projection index storage redesign, vectorized serving generalization, and write-path speedups - #1131
Merged
JohannesLichtenberger merged 31 commits intoJul 21, 2026
Conversation
Documents the planned task #57 resolution: lift projection chunk bytes out of HOT slot values into dedicated CoW-versioned chunk pages keyed from a per-leaf ChunkDirectory, keeping HOT slots tiny. Pins down the as-built interim layout (composite-key 4 KB chunking, slot-0 metadata, PIXC/PIX1/PIXM encodings), the redesign's write/read paths and required commit-walk work, a catalog of invariants to preserve and new hazards (commit-order reference resolution, GC reachability, cache epoch gating), migration via metadata version bump, and the test plan.
…ckDB/ClickHouse Replace byte-offset chunk pages with semantic segments behind the per-leaf directory: record-key segment, one body segment per column (presence bitmap at the head), separate dictionary segments for string columns, and per-segment stats/flags mirrored into the directory entry for I/O pruning. Documents the byte-shift cascade defect of byte-offset chunking over bit-packed encodings, the Parquet/DuckDB/ClickHouse design rules that transfer (and the update stories that deliberately don't), the deliberate 1024-row group divergence, new hazards (fragmentation tax, stats mirror vs segment truth, dict/id cache coherence), and corrects the earlier draft's conflation of per-column HOT slots with per-column segment pages.
…e, phased plan Reviewed the design against the as-built code from four angles (page/commit machinery, query/serving, build/maintenance, float+FSST integration) and corrected every finding: - OverflowPage is a working template for reference-bearing values; adopt its side-map design (refs outside slot bytes, offset identity, write-at- commit key resolution) instead of parsing directories in the commit walk. The hook list is now precise: HOTLeafPage.commit override, writer branch, HOT serializer refs section (sparse-emit correct), fragment-merge carry, PageReference.getPage guard - not combineRecordPages. - Sirix has no page reclamation; the GC hazard is reframed as append-only growth and the migration age-out claim removed. - Slot values become a LeafDescriptor (PIXD) carrying the raw-form header, per-segment byteLen + XXH3-64 contentHash (no-op carry-forward test and integrity check in one), and per-column stats/flags. - Corrected the build-path description (heap-buffered, slot 0 first, putFromSegment is dead code) and scoped update-containment honestly: single-segment rewrites hold for in-place updates only; deletes/appends re-encode the whole leaf; rebuilds become share-friendly. - New hazards from review: live zero-row leaf vs tombstone, sparse dirty-entry emit, dict/id cache coherence (Handle.canonicalDicts), kernel heap-identity constraint (+4.5% MemorySegment regression). - NUMERIC_DOUBLE via ALP and FSST dictionaries promoted from open questions to in-scope (sections 2.6/2.7): order-preserving transform-domain storage so predicate kernels and zone maps work unchanged, NOT_VALUE_EXACT provenance decoupled from long integrality, FSST confined to the persisted DICT segment (kernels compare raw bytes pervasively). - Detailed implementation plan P0-P8 with per-phase files, tests, exit criteria, and a risk register.
…ps, roadmap) New section 8 grounds the OLAP competitiveness intent in the repo's own benchmarks: the 100M DuckDB head-to-head (ahead on 3/9 shapes, 1.1-2.5x on all but count-distinct) and the 10M ClickHouse results; analyzes the scale inversion on dictionary shapes (per-leaf amortization at 97k leaves) and why DICT segments are the prerequisite for fixing it; taxonomizes the gaps (coverage cliff, amortization, type coverage, build cost, native caveats) and maps each to a phase, a follow-up, or explicit out-of-scope; documents the versioned-analytics structural advantage; sequences the post-redesign roadmap (R1 store-level canonical dictionaries, R2 kernel breadth incl. temporal-analytical kernels, R3 morsel-style parallelism); and sets measurable pass/fail bars per milestone in 8.7. P8 gains the benchmark exit artifact (re-run DuckDB protocol, first ClickHouse run at 100M). Sections renumbered 9-11.
ProjectionSegmentPage (PageKind 18, OverflowPage-shaped: opaque bytes, offset identity, leaf of the commit recursion) plus the HOT side-map commit chain from the storage redesign doc (section 2.4): - HOTLeafPage gains a commit override descending into side-map refs, deep-copied refs in copy() (CoW discipline), removePageReference, sorted-key helpers, and split-time ref routing: all four split variants route side-map entries to the page that received their owning slot (owner-slot residency, correct for non-contiguous disc-bit partitions). - HOT_LEAF_PAGE serialization writes a trailing segment-refs section behind a new envelope flag bit; every fragment carries the complete map so the newest fragment is authoritative under fragment merge. Unresolved refs at serialize time fail loudly. New flags-aware envelope helpers keep the strict path for other page kinds. - NodeStorageEngineWriter.commit writes in-memory segment pages and assigns their offset keys inside the commit descent (overflow discipline); resolved refs carried forward are no-ops. - StorageEngineReader.readProjectionSegmentPage default + reader implementation with reference swizzling; writer delegates with in-memory-first resolution. - ProjectionIndexHOTStorage segment API (putSegmentPage, removeSegmentPage, getSegmentPageBytes, readSegmentPageBytes) using the (ownerSlotKey << 8 | segmentId) side-map convention. - ProjectionSegmentPageCommitTest: commit + cold-reopen round trip, deep split cascades with early-attached refs, second-commit sparse fragment merge (replace/carry/remove + time-travel isolation), rollback never writes. Projection, HOT, and page suites green (2706 tests).
20+ candidates from four finder angles, verified and applied: - Side-map key convention promoted to a single contract on HOTLeafPage (segmentRefKey/segmentRefOwnerSlot) with loud validation: segmentId range-checked at every call site (was: put only — remove/get/read silently aliased out-of-range ids onto other segments) and ownerSlotKey checked against the <<8>>8 round-trip (silent collision + sign-extended mis-routing after splits for |key| >= 2^55). - putSegmentPage now requires the owner slot to exist on the target leaf: a ref attached without its owning slot is permanently orphaned after the next split (routing is by owner-slot residency) — durably committed but unreachable, with no error. - removeSegmentPage probes read-only first: the unconditional prepareLeafOfTree CoW'd unchanged leaves into the TIL (spurious fragments per commit) and created a root leaf on an empty trie. - Loud backstops on every uninstrumented leaf-rebuild path that harvests (key,value) pairs and discards the source leaf: subtree rebuild + strand migration + leaf-scoped rebuild (AbstractHOTIndexWriter), incremental split (HOTIncrementalInsert), phase-4 subtree merge and hoist-and-reroute (HOTTrieWriter, hot.strict.binna-gated). All verified unreachable for projection trees today; if wiring ever changes they now fail attributably instead of silently dropping committed segments. - splitTo mid-loop put failure now aborts the split cleanly (target reset, source untouched) instead of truncating the source and dropping the failed entry from BOTH halves. - Segment-page deserialization validates the stored length (corrupt offsets fail as a clean error, not NegativeArraySizeException/OOM); MAX_SEGMENT_BYTES enforced symmetrically at construction. Reader-side resolution type-checks the loaded page (bare offset refs carry no kind tag) instead of blind-casting. - Forwarding reader base class delegates readProjectionSegmentPage (the throwing interface default would surface at the first committed read through any wrapper). - Duplicate writer commit branches merged; KEY_BUFFER reused on the static read path (HFT: no per-call byte[8]); explicit imports per CLAUDE.md; zero-copy no-mutate contracts documented. Projection + HOT + page suites green (2706 tests).
Segment-directory codec layer from the storage redesign doc (section 2.3): - LeafDescriptor: the tiny HOT slot value — raw-form header (rowCount, columnCount, kinds, record-key fences) plus fixed 30-byte entries per segment carrying exact byteLen, XXH3-64 contentHash (write-path no-op comparator and the segments' only checksum), and the per-column stats/flags mirror. Positional allocation-free readers for the scan path; structural validation fails loudly; 84-column cap enforced (8-bit segmentId space). - ProjectionIndexSegmentCodec: splits a raw leaf into semantic segments (KEYS with fences + delta/FOR keys; per-column BODY with flag truth at the head, zone maps, marker-byte presence, FOR/verbatim/packed-id values; per-string-column DICT), each self-describing (PIXS magic + version + kind). assembleRaw reconstructs the raw scan form byte-identically, verifying every segment's length + hash against the descriptor at fill time. Empty leaves emit KEYS + BODYs (flag truth), no DICTs. Encoding primitives shared with ProjectionIndexLeafCodec (widened to package-private, no logic change) - same wire streams, regrouped. - bodySegmentFlags: column-scoped provenance primitive reading segment truth, per the mirror discipline. 15 new tests: byte-identity round trips (bench-shaped 1024-row, empty, single-row, word boundaries, non-ascending + extreme values, 48-64-bit width sweeps, UTF-8 and multi-KB dictionaries, real-empty-string vs phantom), descriptor mirror equals segment truth, deterministic hashes (no-op comparator contract), corrupted/truncated/missing segment fail-loud, column cap.
Applied all 12 verified findings from the two-angle P2 review: - Deduplicated every shared wire authority into ProjectionIndexLeafCodec: string-dict encode split into encodeDictEntries/encodeDictIds halves (used by both the monolithic and segmented codecs), shared decode helpers (decodeForBitPackedColumn, decodeBooleanWords, decodeDictEntries, decodePackedIds, decodePresenceInto) replacing the copy-pasted arms in decode() and assembleRaw, and LeafDescriptor now uses the codec's LE getters. One authority per encoding - drift can no longer break the segmented-vs-monolithic byte-identity guarantee. - Empty-leaf descriptor mirror writes the min>max sentinel pair instead of a fabricated [0,0] range that would defeat descriptor-only pruning (and read as 'possibly contains 0'). - Live-empty-leaf representation reconciled with the design doc: the doc now specifies KEYS + per-column BODY flag-truth carriers (5.1-7) with no DICTs, not segCount=0. - MAX_COLUMNS derived from HOTLeafPage.MAX_SEGMENT_ID and a shared SEGMENTS_PER_COLUMN constant (single authority for the 8-bit id-space invariant); bodySegmentId/dictSegmentId range-check the column so ids can never silently wrap past 255 onto another column's segment. - LeafDescriptor.serialize validates segCount range, parallel-array lengths, and non-negative byteLens (a corrupt length is refused at write time instead of poisoning every later assembly). - encode(null) returns null (absent-leaf symmetry with the monolithic codec); bodySegmentFlags guards the truncated-6-byte-segment case (IllegalStateException, not AIOOBE); EncodedLeaf aliasing contract documented (HFT, no defensive copies). Projection suite green (171 tests).
The segment-directory storage layer (redesign doc sections 2.3/3/4), added alongside the legacy chunked paths until the builder/maintenance/ catalog callers migrate (P4/P5) - the layouts never share a store: - putLeaf: encode via the segment codec, carry forward every segment whose (byteLen, XXH3) matches the prior descriptor entry (CoW share by reference, no page write), write changed/new ProjectionSegmentPages, drop refs of vanished segments (real deletes), store the descriptor as the slot value (plain leafIndex keys). - tombstoneLeaf (zero-length slot + ref removal) vs live empty leaf (rowCount-0 descriptor) - distinct states, both tested. - getLeaf/readLeaf/readAllLeaves: assemble the byte-identical raw form through the side map, tombstone-skipping, ordered enumeration across split topologies; mixed-layout slots fail loudly. - putBlob/getBlob/readBlob: slot-0 metadata path - tiny PIXB marker (byteLen + XXH3) in the slot, whole payload as one segment page, verified on every read (metadata can reach MBs once per-leaf fences scale; tested at 1.5 MB). - segmentPageOffset diagnostic: exposes the durable offset key so tests and P8 measurements can prove sharing, not just byte-equality. Storage-level containment proof (the SLIDING_SNAPSHOT claim, doc 2.5): a single-column edit across two commits shares KEYS + every other BODY + DICT by identical disk offsets and re-writes exactly one BODY page. Shrink-grow-shrink never resurrects stale segments; time travel serves each revision's own bytes. Projection suite green (178 tests).
Applied all 12 verified findings from the two-angle P3 review: - tombstoneLeaf handles blob slots: the PIXB marker failed the descriptor check, so the blob's MB-scale segment ref was never removed - a permanent storage leak carried into every future fragment. Also no-ops on truly absent slots (a tombstone insert CoW'd unchanged leaves and grew the trie for nothing). - putLeaf validates the side-map key precondition BEFORE writing the descriptor slot: failing mid-attach used to leave a poisoned descriptor that a same-trx retry would carry-forward against (hashes match) and skip attaching every segment. - putBlob gained the carry-forward no-op: an unchanged blob (the steady-state metadata case) previously rewrote the full payload page every commit, defeating CoW sharing for the largest payload in the index. Regression test proves sharing by disk offset and the tombstone ref removal. - readAllLeaves fails as loudly as the point reads on mixed-layout slots (silent skipping masked exactly the corruption readLeaf catches) and peeks the magic from the slice before copying (no heap copy for blob/tombstone slots). - updateOrSplitInsert diagnostic is layout-agnostic (raw key + legacy composite interpretation) instead of mis-decoding descriptor slot keys as chunk coordinates. - Write path stops re-hashing carried-forward segments (reads the hashes encode() already stored in the new descriptor); blob marker written via the shared LE writers; navigation preamble and descriptor-guard deduplicated into navigateToSlotLeaf and isLiveDescriptor. Projection + HOT + page suites green (2729 tests).
The persisted projection format flips as one unit — builder, incremental maintenance, catalog reads, creation/drop functions, and the bench setup all move to descriptors + segment pages; ProjectionIndexMetadata.VERSION bumps to 2 so pre-descriptor (chunked) stores degrade to rebuild-on-use per the migration plan (no dual-format reads, no value sniffing). - ProjectionIndexBuilder: truly streaming build - each leaf is written (putLeaf) the moment the builder emits it, one leaf in memory instead of buffering every encoded leaf (~240 MB at 100 M rows); fences accumulate as two longs per leaf; orphaned slots above the new leaf count are tombstoned (real deletes); metadata blob written last. - ProjectionIndexChangeListener: touched-leaf reads via getLeaf (raw form, no decode step), writeLeaf via putLeaf - the hash carry-forward makes untouched-column sharing automatic and full rebuilds share-friendly; metadata refresh and the corruption valve write the slot-0 blob; legacy/corrupt slot-0 payloads degrade to the ladder instead of throwing mid-commit. - ProjectionIndexCatalog: probe via readBlob (legacy stores fall to UNUSABLE -> rebuild), hydrate via readAllLeaves (already raw; serial cursor safe for wtx reads), truncated-store check counts leaves. - CreateProjectionIndex.hydrateLegacy reads the new layout and treats pre-break stores as rebuild-fresh; DropProjectionIndex tombstones via the blob path. - ScaleBenchProjectionSetup persists via putEncodedLeaf (new split of putLeaf exposing encoded sizes without a second codec pass) + metadata blob; probe reads blob + raw leaves. sirix-core projection suite and all sirix-query projection suites green on the new format (incl. ProjectionIndexStressTest soak and the TypedGroupByDifferentialTest byte-identity battery).
All 6 findings from the P4 review, verified and applied: - The v1->v2 migration path actually works now: a rebuild over a legacy (chunked) store RESETS the definition's sub-tree (ProjectionIndexPage.resetProjectionIndexTree + storage.resetTree) - selectively clearing is impossible because composite chunk keys would poison descriptor enumeration with mixed-layout errors forever, permanently un-serving the index while metadata looked valid. Regression test pins reset + fresh build + time-travel to the legacy revision. - priorLeafCount recovers the pre-invalidation leaf count by probing live descriptor slots when metadata is a stale tombstone (the tombstone carries leafCount=0, so re-creates after invalidate/drop left live orphan descriptors above the new count forever). - readAllLeaves enforces the slot-contiguity invariant (5.1-11) loudly - positional hydration against metadata fences would silently mislabel every leaf after a gap - and restores parallel hydrate: phase-2 assembly fans out across the common pool resolving segment pages by durable offsets through throwaway references (no page instances or cursors shared between threads); uncommitted reads and small stores stay serial. The catalog passes its parallelHydrate flag through again. - Bench probe reads the metadata blob first and unconditionally: a catalogued EMPTY projection (leafCount=0) now hydrates instead of being rebuilt every run - the rebuild used to clobber the catalogued definition's metadata with the bench shape, turning the def UNUSABLE. - ProjectionPersistForceRebuildTest migrated to the descriptor layout (it silently exercised the rebuild path instead of hydration). - Metadata VERSION javadoc updated - v1 databases exist and are exactly what the migration gate recovers; the degrade path is load-bearing. Core + query projection suites green; full sirix-core + sirix-query suite run green in background.
… bumps Floating-point projection columns land end-to-end (redesign doc 2.6), decoupled from the long-integrality machinery: - COLUMN_KIND_NUMERIC_DOUBLE stores the sortable-bits involution of the double (negatives flip their low 63 bits), making cells order-isomorphic to signed longs - zone maps, zoneSkip, the numeric predicate kernels (with plan-time-transformed literals), FOR bit-packing, and both codecs work unchanged via kind fall-throughs. ProjectionDoubleEncoding pins the transform with 70k-case order-isomorphism/involution/sentinel property tests. - Extraction converts exactly (Double/Float/Integer widen losslessly; Long round-trip-checked; Big* via exact BigDecimal compare) and reuses the shared provenance bit with kind-dependent semantics: for double columns COLUMN_FLAG_NON_INTEGRAL means 'a stored cell is NOT value-exact', so numericColumnIsIntegral doubles as the provably- value-exact gate with zero format changes. Non-finite values are defensively unrepresentable (no collision with zone-map sentinels). - jn:create-projection-index accepts double/float/decimal; the type mapping routes DEC/DBL/FLO to NUMERIC_DOUBLE (previously silently truncated to longs on non-user paths, rejected on user paths). - conjunctiveAggregateNumericDouble kernel (mask flow shared, two-op decode per matching row) wired through the executor at all three aggregate call sites, surfacing xs:double via Dbl; per-thread partials merge in ascending chunk order for run-to-run-deterministic double summation. Fail-closed gates preserved: not-value-exact or sparse-unclean columns decline to the generic pipeline (parity tests hold either way, by construction). - Metadata VERSION stays 1 per project convention (no deployed databases): the v1->v2 layout migration gate is structural (blob marker check + resetTree), not version-based; doc section 6 updated. Core + query projection suites green (incl. new double end-to-end function tests and the repurposed type-acceptance test).
…scape - DICT segment payloads gain a mode byte: RAW (byte-compatible with the monolithic codec's dictionary half) or FSST - a per-segment symbol table followed by per-entry FSST streams, taken only when the dictionary passes FSSTCompressor's existing gates AND measurably compresses (isCompressionBeneficial). FSST lives in the persisted form only: decode restores plain UTF-8 dictionary bytes, so the raw scan form and every kernel comparing dictionary bytes raw are untouched (doc section 2.7). Training input is the dictionary in interning order - deterministic, so identical re-encodes hash identically and the carry-forward no-op contract (5.2-n) holds, pinned by test. - Tests: high-cardinality URL dictionary round-trips byte-identically and shrinks >2x; escape-heavy (0xFF-laden) adversarial input takes the RAW fallback and round-trips; small dictionaries stay RAW; stale pre-FSST size assertion updated. - FOR numeric decode rejects width bytes above 64 as reserved escapes, making future numeric encodings (ALP for double columns) additive without version machinery - old readers fail attributably instead of misparsing packed bits. Core + query projection suites green.
Predicate extraction assumed long columns: OP_NUM_CMP emitted the raw long literal and OP_FP_CMP rewrote double thresholds into long space - both would compare untransformed literals against the order-preserving transformed cells of a NUMERIC_DOUBLE column and silently match the wrong rows (the kind fall-through in evalNumericBytes made the kernels reachable). Extraction now branches on the column kind: double columns transform the literal (ProjectionDoubleEncoding) and keep the operator (the transform is order-isomorphic, no threshold rewrite), gate on provably-value-exact (a lossy Big*->double cell differs from the source the interpreted pipeline compares exactly), and decline NaN literals and integer literals beyond 2^53. Regression test drives both literal shapes through filtered aggregates with the interpreted pipeline as oracle.
Align the DICT segment codec with KeyValueLeafPage's per-page FSST discipline: parse the symbol table once and run every entry through the parsed-symbol encode/decode overloads. The byte[]-table overloads re-parse the table per entry - O(dictSize x tableLen) pure waste on both the build and hydrate paths. Wire format unchanged (round-trip and determinism tests pin byte-identity).
- DISK_FORMAT.md gains the projection on-disk format (previously undocumented outside class javadoc, a gap the redesign doc called out): descriptor/blob slot values, side-map refs section, segment pages and ids, per-kind segment wire forms, double-transform and legacy-store detection. - KNOWN_LIMITATIONS.md: double-column support entry (value-exactness fail-closed gate, ALP as a reserved escape follow-up) and the legacy-store structural-migration entry. - README projection section: segment-directory storage blurb, double/ decimal column types.
Records what landed on this branch (P0-P4, P5 parity, P6 sans ALP compression, P7, P8 docs), what is deferred with rationale (P5b columnar restructure, ALP behind the reserved escapes, bench-machine scale gates), and renumbers the open questions accordingly.
…ly double aggregates
From the P6 adversarial review:
- CRITICAL: OP_DEC_CMP predicates on double columns compared the
collapsed long-space arm (x > 2.5 => L >= 3) against transformed
cells, silently matching every row ('price > 2.5' is the most natural
literal shape - fractional decimals compile to DecCmp). XQuery
promotes xs:decimal to xs:double against double values, so the
transformed PROMOTED literal with the original operator is exact
parity. Regression test pins the fixed sum (12.125) and count.
- Long.MAX_VALUE exactness edge: its doubleValue rounds up to 2^63 and
the narrowing cast saturates back, so the round-trip check falsely
certified it value-exact (stored double off by one on the very path
whose gate promises exactness). Explicitly lossy now.
- Double-column aggregate serving restricted to count: plain JSON
decimals shred as BigDecimal and the fallback accumulates
decimal-exactly (Dec results), so double-kernel sum/avg/min/max
cannot guarantee digit-and-type parity - the differential oracle
forbids serving them. Predicates and counts serve; the lift path
(pure-double-source provenance bit, additive) is documented in the
redesign doc (11-8) and KNOWN_LIMITATIONS.
- doubleStatsToSequence relocated after longStatsToSequence (javadoc
was misattached), FQN and import-group conventions fixed, stale
'v2 metadata' comment reworded.
The golden format test exists to force conscious acknowledgment of on-disk format additions; the projection segment page (PageKind 18, redesign doc §2.3) is such an addition.
End-to-end walkthrough of the projection index on the segment-directory layout: logical model, page architecture, wire formats (PIXD/PIXS/PIXB/PIXM) with a worked example encoded byte-by-byte, commit chain, hash-based no-op sharing, read path and serving gates, incremental maintenance ladder, double-column transform, FSST dictionaries, migration, and performance positioning — with diagrams throughout.
Make the document approachable for readers who are not storage engineers: a per-audience reading table, a primer on the five foundational ideas (columnar layout, dictionary encoding, zone maps, FOR bit-packing, CoW versioning), and a glossary of the SirixDB-specific vocabulary.
The segment-directory layout (descriptors + segment pages) is the only production format; the chunk machinery had zero production callers. Gone: CHUNK_SIZE and its class-load guard, chunk put/get/readAll and the depth-2 parallel scan, LeafChunkAccumulator, composite chunk keys, the dead off-heap putFromSegment path and serializeIntoSegment, and the two test classes whose subject was that machinery (surviving invariants are covered by ProjectionIndexDescriptorStorageTest, plus a new empty-store test). Migration tests fabricate the pre-redesign layout byte-for-byte through a package-private writeSlotValue seam instead of the removed API. ProjectionIndexHOTStorage: 2072 -> 866 lines; class javadoc now documents the segment-directory contract instead of the interim chunked one.
COLUMN_FLAG_PURE_DOUBLE_SOURCE (flags bit2, additive): set at extraction only when every present double cell came from a Double source — Integer/Long/Big* clear it (the fallback types those Dec), Float clears it (the fallback types Flt and accumulates in float arithmetic). Carried through the PIX1 tail, BODY segments, and the descriptor mirror; probed AND-across-leaves with a fail-closed malformed-payload contract. Under the bit the executor serves sum/avg/min/max over double projection columns; without it, count-only as before. Serving arithmetic reproduces the interpreter bit for bit — adversarial review drove three corrections: - sum/avg use a SEED-FIRST document-order fold: a zero-seeded fold absorbs a lone -0.0 (0.0 + -0.0 == +0.0), and delegation to brackit's aggregator was empirically refuted (its batched double fast path never engages for the document pipeline's non-Dbl items); [1e16, 1, 1] pins the association order. - min/max use Double.compare total order in the kernel (-0.0 < 0.0), matching the interpreter's comparator where IEEE < / > treats them equal. - purity is checked before any leaf work (impure columns decline at metadata cost; count paths skip the probe entirely), and served results are per-func cached ahead of the MixedAgg cache so a hydration-race MixedAgg can never shadow served digits. Also review-driven: resolve-once DoubleAggServing record replaces the acc[4] purity smuggling; reconstruct takes the persisted flags byte verbatim instead of three exploded boolean[]s; probeDoublePureSource keeps all payload reads inside its fail-closed try; null-func guard. New ProjectionDoubleAggregateServingTest pins serving on pure columns, count-only declines for decimal- and integer-fed columns, ill-conditioned sums, and every ±0.0 edge — each against the interpreter's serialized output.
ProjectionAlpEncoding: per leaf-column vector, deterministic two-stage decimal scale-pair selection (32-cell sampled shortlist over the 190 (e,f) candidates, full-vector verification), digits FOR-packed via the shared codec primitives, verbatim exception list (capped at rowCount/8), and a strictly-smaller-than- plain-FOR profitability bar — price-like decimal data drops from ~60-bit transform patterns to ~10-bit digits; non-decimal data falls back to the plain FOR form byte-identically to before. Losslessness is a per-cell proof: every value is verified bit-exact at encode time against the exact decode expression (digits * 10^f) / 10^e — the trailing correctly-rounded DIVISION is load-bearing (a reciprocal multiply misses about half of all k/10^n decimals; the ALP test caught this immediately), and -0.0 routes to the exception list since its digits decode to +0.0. Wire safety: escape 65 is only ever written by the double encoder; 66-255 stay loudly-rejected reserved escapes; an escape byte inside an ALP digits stream is corruption (decodePlainForBitPacked), pinned by a nested-escape test; corrupt scale pairs, exception counts, and rows all reject. Adversarial review passed with two test-coverage gaps, both closed (escape-taken assertion in the -0.0 test; nested-escape rejection case). Determinism end to end keeps the descriptor-hash no-op carry-forward sharing ALP segments across revisions.
sumLiveDescriptorRows walks the projection trie reading only the ~30-byte PIXD slot values — no segment pages — with the same contiguity (5.1-11) and metadata-leafCount truncation checks as a full hydrate, so a descriptor-tier answer can never disagree with what hydration would count. The catalog serves unpredicated record counts through it (countRowsFromDescriptors, own DESCRIPTOR_STATS cache keyed like DATA, enumerated by invalidateUnder and clearCache), and executeAggregate carries a defensive field-less count branch. No current query shape reaches the executor with a field-less count (bare array counts are not pipeline-intercepted), so the catalog API is the load-bearing surface — pinned by a test proving count parity with the hydrate tier while the DATA cache stays empty.
One cache not two (segment-lazy Handle v2 inside DATA, not a sibling column cache); Caffeine fixed-at-insert weights force pessimistic full-projected-size weighing (the win is cold I/O, not resident bytes); segment-scoped kernels migrate one at a time behind the differential suite, numeric/boolean predicates first (string predicates wait for the R1 canonical-dictionary layout).
…dups Serving (projection-index fast paths, always with the generic pipeline compiled alongside as runtime fallback — serving changes cost, never answers): - Per-group aggregates: 1-5 group keys (string and/or integral numeric), count/sum/min/max/avg, first-appearance emission order, composite missing-key components, exact-sum-or-decline overflow policy - Sorted scans: 1-N order keys with per-key direction, stable document-order tiebreaks, lazy per-pull record materialization, heap-based top-K when a sole-consumer fn:subsequence bounds the scan, provably-non-raising let bindings (numeric-proof gated) - Covered-row materialization: record-constructor returns incl. computed +,-,* entries, interpreter-parity empty-sequence semantics for missing fields, eager-materialization row cap - Computed aggregates: sum/avg/min/max/count over +,-,*,idiv,mod trees of covered numeric fields with exact arithmetic (overflow/zero-divisor declines to the interpreter's promoting/raising semantics) - OR predicate trees for all of the above via a shared AND/OR mask evaluator (per-leaf zone downgrades, missing=>false leaf algebra) Correctness hardening from adversarial review rounds: - Extraction-time value-exactness: out-of-range integers and double-typed cells in long columns poison the integrality bit (wrapped longs and double-space folds can no longer serve) - Kind-gated predicate conversion (string-EQ/bool-EBV/numeric arms decline on column-kind mismatch instead of scanning unrelated encodings) - Thread-confined sorted-scan record cursors (per-thread read trx) - Executor-boundary postfix-program validation, per-leaf dict-id and kind-drift guards, packed-key bounds, MAX_GROUP_COLUMNS cap - Fixed Brackit DerefExpr sequence dispatch via SirixDerefExpr port: (pipe).field over parenthesized pipelines evaluated to empty without running the pipe; fresh iterator per iterate() call Write path: - Sticky-winner codec election in the page-body bake-off (probe all codecs every Nth page, run only the elected winner in between; ~10% shred speedup, storage unchanged, probeInterval=1 restores exhaustive pick-smallest) - Per-IndexType most-recent page-container slots (interleaved index streams no longer thrash the shared LRU pair into HashMap lookups) Tests: differential serving suites (parity incl. error outcomes, served/ declined counter proofs), randomized kernel parity vs naive oracles, composition probes for empty-vs-null semantics, overflow/lossy-value/ type-mismatch regression fixtures.
… per-IndexType container slots - Bump io.sirix:brackit to 1.0-alpha9-SNAPSHOT (main + tests artifacts), which carries the upstream DerefExpr sequence-dispatch fix (sirixdb/brackit#109): (pipe).field over parenthesized pipelines now evaluates the pipe and iterates freshly per pass. - Drop the temporary SirixDerefExpr port and its translator override — redundant now that the fix ships in brackit itself. - Sticky-winner codec election for the KEYVALUELEAFPAGE body bake-off: only probe pages (first 8 per thread, then every Nth, -Dsirix.codecBakeoff.probeInterval, default 16) run all three codecs and (re-)elect a winner; pages in between encode with the elected codec alone (~3x less encode work). The codec byte keeps the format self-describing; probeInterval=1 restores exhaustive pick-smallest for byte-identical golden files. Both body writers (template-dedup and inline) now share one emit helper instead of two copies. - Per-IndexType most-recent page-container slots in the storage engine writer: interleaved index streams during shredding no longer thrash the shared two-slot LRU into access-ordered HashMap probes. Slots are stamped only at the post-CoW site (never frozen), and invalidated at snapshot rotation, commit reset, and close.
JohannesLichtenberger
changed the title
The default jn:load/jn:store path on 64-bit Linux/macOS (BasicJsonDBStore defaults to MEMORY_MAPPED) silently fell back to SYNCHRONOUS intermediate auto-commits: the async pre-flush was gated to FILE_CHANNEL, so a 2M-row import stalled the insert thread ~21 times for full serialize+harden cycles (~24% of wall time profiled). Both backends append through the same FileChannelWriter, and a write transaction's storage reads are ordered after the background flush via the flush-permit handoff, so the gate was validation conservatism, not a capability boundary. - Allow KEEP_OPEN_ASYNC_FLUSH on MEMORY_MAPPED (KEEP_OPEN_ASYNC_COMMIT stays FILE_CHANNEL-only until mid-transaction revision publication is validated against remapping mapped readers); document the ordering argument at the guard. - beginImportTrx decides from the RESOURCE's configured storage type, not the store default — per-call overrides to non-capable backends must fall back to sync, never fail the import. - Parallelize the background snapshot flush: double-buffered sliding windows (128 pages) pre-serialize deep copies on a dedicated bounded pool (-Dsirix.asyncFlush.parallelism, default 2 — full fan-out halves insert throughput via memory-bandwidth contention) while the flush thread appends the previous window's cached bytes in snapshot order. A single-threaded flush could not keep pace with rotations, turning the permit backpressure into a near-synchronous stall. - Exempt pages with unresolved overflow references from the background flush: their durable image needs OverflowPage disk keys that only the recursive final commit assigns — flushing froze NULL overflow keys into the stored bytes and silently lost overlong records (>150 KB values). Such pages are promoted back into the live TIL via a new side-channel sentinel and committed with real keys at the end (regression-tested with a 200 KB value crossing an epoch boundary). - Review hardening: parallel-stream leaves capture the first failure instead of throwing (an exceptional root releases joiners while sibling leaves still run — the join must fence every leaf before cleanup scans the windows), close-on-throw for copies that never reach a window, append-pass close ordering, and a high clamp on the parallelism property (an oversized value must degrade, not fail class-init for every write transaction). Semantics: async imports mint ONE revision (no parser-progress checkpoints) and nothing is durable until the final commit; sync mode remains available via -Dsirix.import.asyncFlush=false. The rollback + preallocated-commit offset-reuse interaction is documented in KNOWN_LIMITATIONS. Measured (4-core box, 2M-row / ~22M-node pure-write shred, median of 3, identical binaries): sync 18.1s -> async 15.6s (~14% faster; the flush competes for cores here — wider hosts recover more of the former ~24% commit share). 100M rows: 371.5s vs 428.6s pre-optimization (~13%).
…ex-storage-redesign-i3849v # Conflicts: # bundles/sirix-core/src/main/java/io/sirix/index/projection/ProjectionIndexByteScan.java # docs/PROJECTION_INDEX_STORAGE_REDESIGN.md
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Redesigns the projection-index storage layer and generalizes the vectorized serving layer on top of it, then closes with write-path performance work and a brackit dependency bump. 30 commits, staged with a design doc first and per-phase review rounds.
Storage redesign (P1–P8,
docs/PROJECTION_INDEX_STORAGE_REDESIGN.md)PROJECTION_SEGMENT_PAGE(pinned page-kind id 18) with a HOT side-map commit chain; leaf descriptors (PIXD) + per-segment codecs replace the legacy chunked-values API (now removed).NUMERIC_DOUBLEcolumns via an order-preserving transform, ALP encoding for double BODY streams, FSST-compressed dictionary segments (symbol table parsed once per segment).DISK_FORMAT.mdprojection section,KNOWN_LIMITATIONS.md, a technical deep dive with reading guide and glossary.Serving generalization (projection fast paths in
sirix-query)count/sum/min/max/avg, first-appearance emission order, exact-sum-or-decline overflow policy.fn:subsequencebounds the scan, provably-non-raisingletbindings (numeric-proof gated).+,-,*record entries; computed aggregates over+,-,*,idiv,modtrees with exact arithmetic (overflow / zero divisor declines to the interpreter's promoting/raising semantics).Write path
jn:load/jn:storepath on 64-bit Linux/macOS (store default MEMORY_MAPPED) silently ran ~21 synchronous serialize+harden stalls per 2M-row import because the async pre-flush was gated to FILE_CHANNEL.KEEP_OPEN_ASYNC_FLUSHis now allowed on MEMORY_MAPPED too (both backends append through the sameFileChannelWriter; reads of flushed offsets are ordered by the flush-permit handoff), the import gate consults the resource's actual storage type, and the background flush is parallelized (double-buffered 128-page windows pre-serialized on a small dedicated pool,-Dsirix.asyncFlush.parallelism). Pages whose serialization spills overlong records (>150 KB) into OverflowPages are exempted and promoted to the final recursive commit — flushing them froze NULL overflow keys into durable bytes (silent data loss, caught in adversarial review and regression-tested). Semantics: one import revision instead of parser-progress checkpoints;-Dsirix.import.asyncFlush=falserestores sync mode. Measured on a 4-core box (median of 3, identical binaries): 2M-row shred 18.1s → 15.6s (~14%); 100M rows 371.5s vs 428.6s pre-optimization (~13%) — wider hosts recover more of the former ~24% commit share.-Dsirix.codecBakeoff.probeInterval, default 16) run all three codecs; pages in between encode with the elected winner. The emitted codec byte keeps the format self-describing;probeInterval=1restores exhaustive pick-smallest exactly.IndexTypemost-recent page-container slots inNodeStorageEngineWriter: interleaved index streams during shredding no longer thrash the shared two-slot LRU into access-orderedHashMapprobes.Dependency
io.sirix:brackit1.0-alpha8 → 1.0-alpha9-SNAPSHOT, which carries the upstreamDerefExprsequence-dispatch fix (Fix DerefExpr sequence dispatch and iterator sharing brackit#109):(pipe).fieldover a parenthesized pipeline previously evaluated to empty without running the pipe. The temporary sirix-side port (SirixDerefExpr) is removed now that the fix ships upstream.Related issue
No single tracking issue — this is the follow-on to the projection-index foundation merged in #1116, guided by
docs/PROJECTION_INDEX_STORAGE_REDESIGN.md(first commit of this branch). The brackit half of the dependency change is sirixdb/brackit#109 (merged; deployed to the central snapshots repository by brackit CI).Type of change
Checklist
./gradlew buildpasses locally (JDK 25) — full build is delegated to CI: the local environment cannot fetch the Gradle 9.4.1 wrapper distribution, so verification ran targeted suites on JDK 25 instead (core page/keyed-trie/async-commit/shredder tests,JsonIntegrationTest,ProjectionIndexCatalogServingTest, kernel parity + differential serving suites, sync-vs-async import content-parity incl. an overlong-record epoch-boundary fixture — all green). CI resolves1.0-alpha9-SNAPSHOTfrom the central snapshots repository.docs/formal-verification.mdare still upheld — page/container lifecycle (§5) is unchanged (per-type slots are stamped only at the existing post-CoW site and invalidated at snapshot rotation, commit reset, and close; flush-window copies are closed on every path including failures), and crash recovery (§7) is unaffected (codec election never changes the wire format; async page appends are never referenced until the final commit publishes).pre-commithook)Notes for reviewers
FOAR0001foridiv/modby zero, overflow promotion to decimal).CHANGELOG.md): default Linux/macOS bulk imports now mint ONE revision and nothing is durable until the final commit — the sync mode's parser-progress checkpoint revisions (and their crash-survivability) are available via-Dsirix.import.asyncFlush=false. The rollback +-Dsirix.commit.preallocated=trueoffset-reuse interaction is documented inKNOWN_LIMITATIONS.md.probeInterval > 1the codec chosen for a page depends on per-thread serialization history, so stored bytes are not a pure function of page content (content round-trips identically; sizes may differ by a few bytes run to run). Golden-file byte comparisons must pin-Dsirix.codecBakeoff.probeInterval=1.KNOWN_LIMITATIONS.md/ the deep dive): cross-field OR predicates are not yet annotated for serving by brackit's detection (single-field OR trees are); very large stores (100M-record scale) currently decline group-aggregate serving — under investigation, generic fallback semantics unaffected.