[Feature-request] [BigQuery Destination] Allow Custom Partitioning with Full Refresh #74060
Jeroen Maagdenberg (jmaagdenbergSB)
started this conversation in
Connector Ideas and Features
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Context
I am currently running a daily Full Refresh sync on a large dataset spanning several years. Because the destination table is not partitioned by the business logic date (
event_date), but rather by the default_airbyte_extracted_at(which is redundant in a Full Refresh scenario), downstream compute costs are excessively high.Proposed Change
Allow users to specify a custom partition column (e.g.,
event_date) within the destination connector configuration, rather than defaulting to the Airbyte metadata columns. (at least for the Full Refresh).Beta Was this translation helpful? Give feedback.
All reactions