feat(aws_s3 sink): Add Apache Parquet encoder support

[rorylshanks]

Summary

This PR adds Apache Parquet encoding support to the AWS S3 sink, enabling Vector to write columnar Parquet files optimized for analytics workloads.

Parquet is a columnar storage format that provides efficient compression and encoding, making it ideal for long-term storage and query performance with tools like AWS Athena, Apache Spark, and Presto. This implementation allows users to write properly formatted Parquet files with configurable schemas, compression, and row group sizing.

Key features:

• Complete Parquet encoder implementation with Apache Arrow integration
• YAML schema configuration support (field names → data types)
• Support for all common data types (strings, integers, floats, timestamps, booleans, etc.)
• Configurable compression algorithms (snappy, gzip, zstd, lz4, brotli)
• Row group size control for query parallelization
• Nullable field support
• Comprehensive test suite (9 unit tests)
• Full documentation for schema configuration and Parquet options

Vector configuration

sources:
  events:
    type: kafka
    bootstrap_servers: "kafka:9092"
    topics:
      - events

transforms:
  prepare:
    inputs:
      - events
    type: remap
    source: |
      parsed = parse_json(.message) ?? {}
      .uuid = parsed.uuid
      .properties = parsed.properties
      

sinks:
  s3_events:
    type: aws_s3
    inputs:
      - prepare
    bucket: my-bucket
    region: us-east-1
    compression: none  # Parquet handles compression internally

    batch:
      max_events: 50000
      timeout_secs: 60

    encoding:
      codec: parquet
      parquet:
        compression: zstd
        allow_nullable_fields: true
        schema:
          timestamp: timestamp_microsecond
          uuid: utf8
          properties: utf8

How did you test this PR?

I tested it against production Kafka data, and it produced correctly formatted Parquet files in S3.

Change Type

• Bug fix
• New feature (Parquet encoder for AWS S3 sink)
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: Support parquet columnar format in the aws_s3 sink #1374
• Closes: enhancement(aws_s3 sink): Add parquet codec #17395

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[rorylshanks]

I have read the CLA Document and I hereby sign the CLA

[thomasqueirozb]

Hey @rorylshanks, thanks for your contribution! It looks like there are failing checks (run make check-clippy for example). This is also failing to compile after merging master because you removed BatchSerializerConfig::build which is used by the clickhouse sink. I'll circle back to this PR and give it a review once I see commits pushed to this branch

[rorylshanks]

Hey Thomas! Thanks for the review so far. I have made some changes to fix the check issues, and also to change how the docs are templated, as I think this codec only makes sense for AWS S3 and other S3 APIs, and I don't really have time to test that the parquet implementation works for every sink. So I limited the docs to only include the parquet format for S3 et al, and not for others. But please correct me if this is the wrong approach!
I am currently having issues with the cue docs checker, as I have changed how the templating works for the docs, the check is not happy, and I can't figure out how to make it happy. Do you have any ideas here?

[abhishek-dream11]

Hi @rorylshanks , is buffer supported added for back pressure handling ? nice for taking this up.

[thomasqueirozb]

Is there any particular reason for this? I assume it'd be possible to use the parquet codec with the file sink also no? In general we don't "restrict" codecs like this, sometimes they are incredibly useful for debugging. (One recent example that comes to mind is the otlp codec which is not useful at all outside of some select sinks, notably the opentelemetry sink, but we still support it for all sinks)

[thomasqueirozb]

Suggested change

	1. **Explicit Schema**: Define the exact structure and data types for your Parquet files
	2. **Automatic Schema Inference**: Let Vector automatically infer the schema from your event data
	1. **Explicit Schema**: Define the exact structure and data types for your Parquet files.
	2. **Automatic Schema Inference**: Let Vector automatically infer the schema from your event data.

[thomasqueirozb]

Suggested change

	### allow_nullable_fields
	When enabled, missing or incompatible values will be encoded as NULL even for fields that
	would normally be non-nullable. This is useful when working with downstream systems that
	can handle NULL values through defaults or computed columns.

This can be removed as we already have the automatically generated doc for this option

[thomasqueirozb]

This info should be present in the automatically generated documentation for these fields instead

[thomasqueirozb]

Is there a .parquet format (or something similar) so that the user can point to a file instead of defining this inside the configuration yml? In these scenarios we usually opt for a config file instead. See parse_proto and validate_json_schema.

[thomasqueirozb]

Same comment

[thomasqueirozb]

In general we should avoid including parameter descriptions and examples in this file. These should live in source code instead so that the documentation is easier to maintain and avoid drift

[thomasqueirozb]

Suggested change

	tracing::debug!(
	debug!(

Nit: we usually avoid using tracing:: and opt to import this at the start of the file instead.

[thomasqueirozb]

Please use a static constant instead. Magic values tend to fall out of date with docs

[thomasqueirozb]

I think that using #[configurable(metadata(docs::examples = "foo bar"))] works here. I'm not sure how this would render with multi line examples but if it the rendering doesn't work properly I'll make sure to fix it in a separate PR

[thomasqueirozb]

This can be expressed with an enum which will simplify a lot of the checks. It will also render docs in such a way that will make it clear when certain options are available with a schema and those available without.

Something like

enum Schema {
  Inferred { infer_schema: bool, exclude_columns: Option<Vec<String>> },
  Schema { schema: SchemaDefinition },
}


/// Configuration for Parquet serialization
#[configurable_component]
pub struct ParquetSerializerConfig {
  #[serde(flatten)]
  schema: Schema

This is very rough and removing all doc comments but should work. I'm sure you can find examples of how to do this properly in the Vector code base

[thomasqueirozb]

I have made some changes to fix the check issues, and also to change how the docs are templated, as I think this codec only makes sense for AWS S3 and other S3 APIs, and I don't really have time to test that the parquet implementation works for every sink.

I wouldn't expect anyone to test every sink! I think we can do just a quick sanity check and write to a file using the file sink or event a socket using the socket sink and verify that it is working correctly. We can then add a note to the codec docs specifying which sinks it should be used with. As I mentioned before, we usually show all codecs, even if it is very unusual for anyone to use some of them.

I am currently having issues with the cue docs checker, as I have changed how the templating works for the docs, the check is not happy, and I can't figure out how to make it happy. Do you have any ideas here?

don't worry about this for now. Worst case scenario I can fix this myself before merging :)