[PLUGIN-1931] Add defaults for Time and Range Partitioning in BQ source plugin#1583
Merged
adrikagupta merged 1 commit intodevelopfrom Sep 25, 2025
Merged
[PLUGIN-1931] Add defaults for Time and Range Partitioning in BQ source plugin#1583adrikagupta merged 1 commit intodevelopfrom
adrikagupta merged 1 commit intodevelopfrom
Conversation
1d9e162 to
21de8fc
Compare
21de8fc to
1d4aa48
Compare
src/main/java/io/cdap/plugin/gcp/bigquery/source/PartitionedBigQueryInputFormat.java
Outdated
Show resolved
Hide resolved
src/test/java/io/cdap/plugin/gcp/bigquery/source/BigQuerySourceConfigTest.java
Outdated
Show resolved
Hide resolved
src/main/java/io/cdap/plugin/gcp/bigquery/source/BigQuerySourceConfig.java
Outdated
Show resolved
Hide resolved
src/main/java/io/cdap/plugin/gcp/bigquery/source/BigQuerySourceConfig.java
Outdated
Show resolved
Hide resolved
src/main/java/io/cdap/plugin/gcp/bigquery/source/PartitionedBigQueryInputFormat.java
Show resolved
Hide resolved
1d4aa48 to
85633f7
Compare
src/main/java/io/cdap/plugin/gcp/bigquery/source/BigQuerySourceConfig.java
Show resolved
Hide resolved
itsankit-google
approved these changes
Sep 25, 2025
Contributor
itsankit-google
left a comment
There was a problem hiding this comment.
one small comment, otherwise LGTM
This was referenced Oct 1, 2025
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If we pass a filter input in BQ source plugin and keep partitionFromDate and partitionToDate values as empty for a table where requirePartitionFilter is set as true, then the preview/deployment run will fail because since requirePartitionFilter is true, the query requires a mandatory “where” clause on partitioning column.
Different types of partitioning for BQ are listed in https://cloud.google.com/bigquery/docs/partitioned-tables .
To fix we are adding defaults for range, time unit and ingestion partitioning which would be used if requirePartitionFilter is set as true and no partitioning inputs are passed. The default would be "parititioning-column is NOT NULL OR parititioning-column is NULL" to get the entire data.
A few things were noted when considering these defaults:
Also, we have added limit and order by fields in UI since filter field is only meant for WHERE clause.
Testing: