-
Notifications
You must be signed in to change notification settings - Fork 2k
Add RFC for MaterializePartitionColumns #5556
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 4 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,33 @@ | ||||||
| # Materialize Partition Columns | ||||||
|
|
||||||
| **Associated Github issue for discussions: https://github.com/delta-io/delta/issues/5555** | ||||||
|
|
||||||
| ## Overview | ||||||
|
|
||||||
| Currently, Delta tables store partition column values primarily in the table metadata (specifically in the `partitionValues` field of `AddFile` actions), and by default these columns are not physically written into the Parquet data files themselves. | ||||||
|
|
||||||
| This RFC proposes a new writer-only table feature called `materializePartitionColumns`. When enabled, this feature requires partition columns to be physically materialized in Parquet data files alongside the data columns. | ||||||
|
|
||||||
| ## Motivation | ||||||
|
|
||||||
| This feature provides a mechanism to require partition column materialization at the protocol level, ensuring all writers to the table comply with this requirement during the period when the feature is enabled. | ||||||
|
|
||||||
| Materializing partition columns enhances compatibility with Parquet readers that access Parquet files directly and do not interpret Delta’s AddFile metadata, as well as with Iceberg readers, which expect partition columns to be stored within the data files. | ||||||
|
|
||||||
| Additionally, having partition information embedded in the data files themselves enables more flexible data reorganization strategies, as files can be physically rearranged without strict partition directory constraints while still maintaining partition information. | ||||||
|
|
||||||
| -------- | ||||||
|
|
||||||
|
|
||||||
| > ***New Section after Identity Columns section*** | ||||||
| ## Materialize Partition Columns | ||||||
|
|
||||||
| When this feature is enabled, partition columns are physically written to Parquet files alongside the data columns, which can improve flexibility with respect to data layout changes in the future, and make these data files easier to interpret for readers unfamiliar with partition values. To support this feature: | ||||||
| - The table must be on Writer Version 7, and a feature name `materializePartitionColumns` must exist in the table `protocol`'s `writerFeatures`. | ||||||
|
|
||||||
| When supported: | ||||||
| - The table respects metadata property `delta.enableMaterializePartitionColumnsFeature` for enablement of this feature. The writer feature `materializePartitionColumns` is auto-supported when this property is set to `true`. | ||||||
|
||||||
| - The table respects metadata property `delta.enableMaterializePartitionColumnsFeature` for enablement of this feature. The writer feature `materializePartitionColumns` is auto-supported when this property is set to `true`. | |
| - The table respects metadata property `delta.enableMaterializePartitionColumnsFeature` for enablement of this feature. The writer feature `materializePartitionColumns` is auto-enabled when this property is set to `true`. |
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's call out here that this feature mimics this same requirement from IcebergCompat* (linking all the IcebergCompat features that have a similar requirement), and that this means that this feature can be seen as a subset of the requirements of those feature(s)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added that IcebergCompatV1 / IcebergCompatV2 have the same requirement.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| This feature does not impose any requirements on readers. All Delta readers must be able to read the table regardless of whether partition columns are materialized in the data files. If partition values are present in both parquet and AddFile metadata, readers should continue to read partition values from AddFile metadata. | |
| This feature does not impose any requirements on readers. All Delta readers must be able to read the table regardless of whether partition columns are materialized in the data files. If partition values are present in both parquet and AddFile metadata, Delta readers should continue to read partition values from AddFile metadata. |
mostly just disambiguates this from non-delta parquet readers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's remove this part because the directory structure is not at all required by the protocol. Instead we can say "the same parquet files could be linked in future versions of a table that do not have the same (or any) partition columns".