Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR:
SchemaWithPartner
visitor to refactorRecordBatchProjector
RecordBatchProjector
inRecordBatchTransform
Context
For read or write in iceberg-rust, we interoperate with arrow RecordBatch. There are similar project tasks when reading and write to project RecordBatch:
For now, these tasks are implemented in different structs,
RecordBatchTransform
, andRecordBatchProjector
and there are redundant codes. After we introduce theSchemaWithPartner
visitor in #731, we can unify the implementation under the "visitor" framework.The project task is generally separated into two parts:
We can custom accessor to achieve init-default. E.g. when fail to retrieve the array, accessor can construct the array according to init-default. And we also can custom accessor to achieve name-map.
We can customize visitors to do the type of promotion here.
The PR try this work to refactor our
RecordBatchTransform
, andRecordBatchProjector
. And it also complete #405 (comment) using ArrowProjectionVisitor as @liurenjie1024 mention