Skip to content

Pravega Flink Connectors 0.9.0
Compare
Choose a tag to compare
@crazyzhou crazyzhou released this 09 Mar 10:17
· 107 commits to master since this release
v0.9.0
16c6b94

This release contains exciting new features and significant enhancements over the previous major release line (0.8.x). This page provides an overview of major ones.

This release supports Flink 1.9, 1.10 and 1.11, and it is tested and validated with Pravega 0.9.0 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector

Highlight features

  • (#392) FLIP-95/FLIP-122 new Flink Table API in Flink 1.11

In the latest Flink 1.11 connector, new Table API described in FLIP-95 is supported, a simpler CREATE TABLE SQL DDL can be used to create the table-stream connection instead of the descriptor style for legacy Table API. e.g.

CREATE TABLE test (
    user_id STRING,
    item_id BIGINT)
with (
    'connector' = 'pravega'
    'controller-uri' = 'tcp://localhost:9090',
    'scope' = 'scope',
    'scan.execution.type' = 'streaming',
    'scan.reader-group.name' = 'group1',
    'scan.streams' = 'stream',
    'sink.stream' = 'stream',
    'sink.routing-key.field.name' = 'user_id',
    'format' = 'json'
)

Introduction with more details can be found in the official document.

  • (#422) Add schema registry support for streaming and batch reader

The connector can integrate with Pravega schema registry, and users can use below method to use the schema registry deserializer to read events from streams

// e.g. For avro generic record in streaming source,
FlinkPravegaReader.<GenericRecord>builder()
     ....
     .withDeserializationSchemaFromRegistry(streamName, GenericRecord.class)
     .build()
  • (#426) Upgrade to Java 11

The compatibility for Java 8 is still kept, so it is still fine to use the artifact under Java 8 environment. The main difference is when developing this project and running the unit tests, Java 11 is required.

Notable bugfixes

  • (#377) Refactor Pravega Sink to extend TwoPhaseCommitSinkFunction
  • (#433) Make event router optional
  • (#455) Increase the default transaction lease renewal time to 120 seconds

Notifications

  • (#438) Legacy table API is deprecated, please upgrade to Flink 1.11 connector and use the new Table API.
  • (#439) Deprecated PravegaSerialization is removed, please use the serialization/deserialization schema instead.

Flink 1.11 with Scala 2.12 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.log

Assets 14
Loading