[Feature][Connector-V2] Support databend source/sink connector

[hantmac]

Purpose of this pull request

Add the Databend connector. Fix #9315

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Add UT and e2e tests.

Check list

• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update plugin-mapping.properties and add new connector information in it
  2. Update the pom file of seatunnel-dist
  3. Add ci label in label-scope-conf
  4. Add e2e testcase in seatunnel-e2e
  5. Update connector plugin_config

[Copilot]

Pull Request Overview

Add a new Databend connector module, providing both source and sink implementations for JDBC-based reads and writes to Databend.

• Introduces DatabendSource and DatabendSink with factories, reader/writer, state, and utility classes
• Supplies example HOCON configs and updates plugin_config, plugin-mapping.properties, and distribution POM
• Adds documentation (README.md) and service registration for auto-discovery

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
databend_source_example.conf	Example HOCON for source usage
databend_sink_example.conf	Example HOCON for sink usage
META-INF/services/.../Factory	Registers Databend source and sink factories
DatabendUtil.java	JDBC connection, type conversion, SQL generation
DatabendSourceFactory.java / DatabendSinkFactory.java	Factory rules and plugin identifiers
DatabendSourceReader.java / DatabendSinkWriter.java	Runtime reader and writer logic
README.md	Connector documentation with usage examples
config/plugin_config	Adds connector-databend to plugin config

[hailin0]

Thanks @hantmac

Please update docs

[hantmac]

Thanks @hantmac

Please update docs

@hailin0 Thanks! I have added the docs. PTAL.

[Hisoka-X]

I notice dataend read/write data use jdbc, why not add new dialect in connector-jdbc?

[hantmac]

I notice dataend read/write data use jdbc, why not add new dialect in connector-jdbc?

@Hisoka-X We want to maintain extensibility, we plan to add streaming upload as a data import method to Databend later. Also, we can use replace into and merge into to impl CDC feature in later PR to make databend sink much more strong.

[Hisoka-X]

Please follow the guide to open ci on your fork reposity. https://github.com/apache/seatunnel/pull/9331/checks?check_run_id=43433559569

[hantmac]

hi @Hisoka-X , there are some errors in CI in some other module and it seems that the tests of databend connector module not running?

[Hisoka-X]

Hi @hantmac . Databend e2e already started in https://github.com/hantmac/seatunnel/actions/runs/15437221086/job/43446415225. Please fix the e2e test error.

The unit test have some test also failed.

[Hisoka-X]

Other errors may be caused by instability and can be resolved by retrying.

[hantmac]

hi @Hisoka-X , sorry to bother you, I run the databend-e2e and ut locally and pass all the tests.

Why in the ci I always got condition timeout error https://github.com/hantmac/seatunnel/actions/runs/15482805214/job/43591670655#step:5:31657?

[Hisoka-X]

We have a feature called thread leak check for connector, which checks for threads that have not been released in the connector. It seems that there are threads that are delayed in releasing in databend, which causes this error. It's okay, maybe we can adjust the timeout to 120 seconds, just like you did at the beginning. https://github.com/apache/seatunnel/pull/9331/files#diff-828bb95593582c2297e60ef9a6ced4733f251eef77389235d8c3b506dd0beb47R343

[hantmac]

adjust the timeout

Got it. Thanks!

[hantmac]

Hi @Hisoka-X , I check the test CI logs and it seems that the tests of databend maybe have past successfully.

And What need I do in the next step?

[Hisoka-X]

Could you merge TestDatabendCase into DatabendIT? We use name to match it is unit test or not. Start with Test class is unit test. So the unit test failed now. https://github.com/hantmac/seatunnel/actions/runs/15484736824/job/43596998748

[hantmac]

Thanks! I have merge TestDatabendCase into DatabendIT and pass the e2e locally.

[Copilot]

Pull Request Overview

This pull request adds a new Databend connector for both source and sink, addressing issue #9315. The changes include the implementation of connector classes, configuration options, exception handling, schema management, catalog factory integration, documentation updates, and necessary CI/pom modifications.

Reviewed Changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 2 comments.

File	Description
connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/sink/DatabendSink.java	Implements the Databend sink with type conversion and writer creation.
connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/schema/SchemaChangeManager.java	Provides schema evolution handling for Databend tables.
connector-databend/src/main/java/org/apache/seatunnel/connectors/seatunnel/databend/config/*	Adds configuration and options classes for source and sink.
Other files (exceptions, catalog, pom.xml, plugin-mapping.properties, docs, workflows)	Update ancillary configurations, documentation and CI definitions to support the new connector.

[Hisoka-X]

waiting test case passes.

[hantmac]

waiting test case passes.

@Hisoka-X Is there any test case still failed?

[Hisoka-X]

waiting test case passes.

@Hisoka-X Is there any test case still failed?

We need all test case passes before merge. So if other test case failed please retrigger failed ci.

[hantmac]

waiting test case passes.

@Hisoka-X Is there any test case still failed?

We need all test case passes before merge. So if other test case failed please retrigger failed ci.

Hi bro @Hisoka-X , I make all the tests pass finally. What need todo in the next step

[Hisoka-X]

Thanks @hantmac