[FLINK-35936][cdc-connector][paimon] Fix paimon cdc schema evolution failure when restart job

[MOBIN-F]

paimon cdc schema evolution failure when restart job

Minimal reproduce step：

1. stop flink-cdc-mysql-to-paimon pipeline job
2. alter mysql table schema, such as add column
3. start pipeline job
4. the newly added column was not synchronized to the paimon table

[leonardBang]

Thanks @MOBIN-F for the contribution, could you add description for the PR ?

[lvyanquan]

So the expected scenario to be solved is actually a scenario where the table structures of downstream that was previously created are inconsistent with upstream(mysql)?

[MOBIN-F]

Yes

[lvyanquan]

I think this change carries risks, I tend to add a sink option to explicitly support this:
When the upstream and downstream table structures are inconsistent, users can manually modify the table structure to adapt.
If this is done in the program, in the case where the user configures the wrong route rule (which is not a particularly rare situation), the situation where the table structure does not correspond should throw an exception. However, we will forcibly modify downstream tables to adapt, and the user will not be aware of this error.

[MOBIN-F]

thanks, You are right, I only considered the * situation before

[MOBIN-F]

@lvyanquan Could you please review it again? thank you

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.

[github-actions]

This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.

[github-actions]

This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.

[lvyanquan]

Hi @MOBIN-F.
I think that this can be closed as it was solved in #4081.
I believe that modifying the data to adapt to the downstream table structure is a safer solution compared to modifying the downstream table structure through the task to accommodate the data. What do you think?

[MOBIN-F]

Hi @MOBIN-F. I think that this can be closed as it was solved in #4081. I believe that modifying the data to adapt to the downstream table structure is a safer solution compared to modifying the downstream table structure through the task to accommodate the data. What do you think?

agree~