[persist] PARTITION BY for Persist-backed collections #30355
Conversation
bkirwi
force-pushed
the
partition-by
branch
4 times, most recently
from
3e38745 to
cc87aae
Compare
bkirwi
changed the title
| /// Returns true iff the ordering of the "raw" and Persist-encoded versions of this columm would match: | ||
| /// ie. `sort(encode(column)) == encode(sort(column))`. This encoding has been designed so that this | ||
| /// is true for many types. | ||
| pub fn preserves_order(scalar_type: &ScalarType) -> bool { |
There was a problem hiding this comment.
Is there anyway to add tests for this to make sure it's correct? Also when adding a new type, how do we know whether to return true or false?
There was a problem hiding this comment.
Good callout! I've added some lines to our property test that checks the sorted column is also sorted according to the schema when preserves_order is true.
Also when adding a new type, how do we know whether to return true or false?
I think the idea is that whoever is designing a new encoding for a type should make a reasonable effort to ensure this property is true... so they should be in a good position to know whether they managed it or not. (And hopefully now the test will keep them honest!)
|
Merging to avoid rot - if anyone has late-breaking feedback I'll make sure to fold it in for the next round. |
Add a new
PARTITION BYclause on materialized views. (Behind a flag!)This has no effect besides asserting that the new "structured" ordering we're using for Persist data will match the SQL-level ordering on those columns. See the design doc for more details.
Motivation
Issue: https://github.com/MaterializeInc/database-issues/issues/7188
Draft design: https://www.notion.so/materialize/PARTITION-BY-13413f48d37b8099aaaac45902ffdb75
Tips for reviewer
This is prototype syntax, behind a flag! Please direct thoughts on the supported syntax to the design doc - we're collecting opinions there.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.