Pravega Flink Connectors 0.10.1
This release contains exciting new features and significant enhancements over the previous major release line (0.9.x). This page provides an overview of major ones.
This release supports Flink 1.11, 1.12 and 1.13, and it is tested and validated with Pravega 0.10.1 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Highlight features
The connector fixes the compatibility issues and updates its API to adapt Flink 1.12 and Flink 1.13.
- (#424) Pravega Catalog support
In the latest Flink 1.12 and 1.13 connector, Catalog API, also known as FLIP-30, is implemented in the connector with the help of Pravega and its Schema Registry. Schema registry provides Pravega stream metadata such as schema and serialization that the connector can accommodate and then present in Flink as a typical database catalog. Once the schemas are registered and the Flink catalog is set, Pravega streams are automatically mapped into tables without the CREATE TABLE SQL DDL.
To create such a catalog, just have the schema registry set up and try with CREATE CATALOG DDL statement with the options listed here.
- (#530) Table API Debezium support
Since Debezium 1.6.0 release, Pravega has been one of the supported sink for Debezium server, and Flink can act as a query engine to perform continuous queries with the ever-changing table. After the Flink 1.13 connector implements the multiple rows emission in deserialization, users can now use the official Flink debezium-json format to consume the CDC data in Pravega streams.
- (#494) FLIP-107: Expose metadata for new Table source
With FLIP-107, Flink has added the support for metadata along with the event for both table source and sink. Pravega Flink connector integrates the API to export metadata from both the connector and format. Furthermore, the EventPointer in bytes is provided as metadata in the table source.
- (#571) Add Python datastream API support
Python DataStream API connector is now integrated as an experimental feature. The basic idea is to use py4j to call the Java implementation. The suggested implementation for such a python wrapper is offered in the code base. Users can include the python files in the project and follow the guide to develop the pyflink application.
- (#445) Adopt github actions for CI pipeline
We have transitioned our CI pipeline from Travis to Github Actions to have it more stable.
Notable bugfixes
- (#441) Fail to reset reader group with no checkpoint recovery
- (#517) Fix NPE issue for recovery job after enabling transactions
- (#481) Row-based router cannot be specified on non-null field
Notifications
- (#556) Drop support for Scala 2.11
Flink community is trying to drop support for Scala 2.11 for some reasons (Detail in: https://issues.apache.org/jira/browse/FLINK-20845) and we also receive some feedback that Scala 2.11 is less used. We have decided to no longer release the Scala 2.11 artifacts for the connector. For users still on Scala 2.11, it's now required to build the connector on your own to use.
- (#493) Make reader group optional in table API
If the reader group name is set in Flink table API, the table cannot be reused as duplicate createReaderGroup calls would mess up. It is suggested not to use this scan.reader-group.name option and the connector can give a random one so the Flink table can be reused.
- (#490) Use maven publish instead of maven plugin in gradle
The Gradle maven plugin is deprecated according to the offical website and is recommended to use the new maven-publish plugin. Developers need to pay attention to the packaging script changes accordingly.
Flink 1.13 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt