feat(datafusion): Support insert_into in IcebergTableProvider #1511
Conversation
| @@ -440,10 +440,12 @@ impl PartnerAccessor<ArrayRef> for ArrowArrayAccessor { | |||
| Ok(schema_partner) | |||
| } | |||
|
|
|||
| // todo generate field_pos in datafusion instead of passing to here | |||
There was a problem hiding this comment.
I found it tricky to handle this case: the input from datafusion won't have field id, and we will need to assign them manually. maybe there is a way to do name mapping here?
There was a problem hiding this comment.
Thanks @CTTY for this pr, just finished round of review. My suggestion is to start with unpartitioned table first.
| // Define a schema. | ||
| Arc::new(ArrowSchema::new(vec![ | ||
| Field::new("data_files", DataType::Utf8, false), | ||
| Field::new("count", DataType::UInt64, false), |
There was a problem hiding this comment.
What's the meaning of count?
There was a problem hiding this comment.
Datafusion expects insert_into to return the number of rows(count) it written: https://datafusion.apache.org/user-guide/sql/dml.html#insert Here I'm sending count to the commit node, and have the commit node to return the number of rows eventually.
Technically we don't need to follow Datafusion's convention on insert_into and can return nothing, do you think that would be better?
There was a problem hiding this comment.
I think we should still follow datafusion's convention. But do we really need this? DataFile has a field called record_count, and I think it's enough for insert only case?
There was a problem hiding this comment.
Yeah using record_count makes more sense, I'll fix this
| @@ -432,3 +433,69 @@ async fn test_metadata_table() -> Result<()> { | |||
|
|
|||
| Ok(()) | |||
| } | |||
|
|
|||
| #[tokio::test] | |||
| async fn test_insert_into() -> Result<()> { | |||
There was a problem hiding this comment.
I'm not a big fan of adding this kind of integration tests. How about adding sqllogictests?
| // Define a schema. | ||
| Arc::new(ArrowSchema::new(vec
Which issue does this PR close?
What changes are included in this PR?
Are these changes tested?