iceberg: add configurable Parquet string encoding for Redshift Spectrum compatibility#4401
Conversation
|
Commits Review LGTM |
b572717 to
bc5417a
Compare
|
Commits Review LGTM |
58f3a1d to
77d7709
Compare
|
Commits Review LGTM |
| // Parse parquet config | ||
| var writerOpts []parquet.WriterOption | ||
| if conf.Contains(ioFieldParquet) { | ||
| strEnc, err := conf.FieldString(ioFieldParquet, ioFieldParquetStringEncoding) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("parsing %s: %w", ioFieldParquetStringEncoding, err) | ||
| } | ||
| switch strEnc { | ||
| case "plain": | ||
| writerOpts = append(writerOpts, parquet.DefaultEncodingFor(parquet.ByteArray, &parquet.Plain)) | ||
| case "delta_length_byte_array": | ||
| // default - noop | ||
| default: | ||
| return nil, fmt.Errorf("unsupported %s value: %q, please consider raising an issue to request support for feature gap", ioFieldParquetStringEncoding, strEnc) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
This new parquet.string_encoding config field is introduced without any test coverage. Per the project's test patterns, changed code should be accompanied by tests — at minimum a config-parsing/linting test exercising both plain and delta_length_byte_array (and ideally a writer test verifying the option is actually plumbed through to the underlying parquet writer). The existing writer_test.go and router_test.go could be extended for this.
|
Commits
Review Adds a configurable Parquet
|
Uh oh!
There was an error while loading. Please reload this page.