Skip to content

[PLUGIN-1931] Add defaults for Time and Range Partitioning in BQ source plugin#1583

Merged
adrikagupta merged 1 commit intodevelopfrom
fix-bq-source-partitioning
Sep 25, 2025
Merged

[PLUGIN-1931] Add defaults for Time and Range Partitioning in BQ source plugin#1583
adrikagupta merged 1 commit intodevelopfrom
fix-bq-source-partitioning

Conversation

@adrikagupta
Copy link
Contributor

@adrikagupta adrikagupta commented Sep 23, 2025

If we pass a filter input in BQ source plugin and keep partitionFromDate and partitionToDate values as empty for a table where requirePartitionFilter is set as true, then the preview/deployment run will fail because since requirePartitionFilter is true, the query requires a mandatory “where” clause on partitioning column.
Different types of partitioning for BQ are listed in https://cloud.google.com/bigquery/docs/partitioned-tables .

To fix we are adding defaults for range, time unit and ingestion partitioning which would be used if requirePartitionFilter is set as true and no partitioning inputs are passed. The default would be "parititioning-column is NOT NULL OR parititioning-column is NULL" to get the entire data.
A few things were noted when considering these defaults:

  • Time-unit partitioning columns can have future-dated values.
  • We are not using startDate and endDate for range partitioning, as it's possible to have values that do not lie within the specified range.

Also, we have added limit and order by fields in UI since filter field is only meant for WHERE clause.

Testing:

  • Created a pipeline with BQ source plugin. Used a table having requirePartitioningFilter as true and ingestion partitioning. Provided filter as "1=1", order by and limit fields. Preview and deployment run was successful.
  • Created a pipeline with BQ source plugin. Used a table having requirePartitioningFilter as true and time-unit partitioning. Provided filter as "1=1", order by and limit fields. Preview and deployment run was successful.
  • Created a pipeline with BQ source plugin. Used a table having requirePartitioningFilter as true and range partitioning. Provided filter as "1=1", order by and limit fields. Preview and deployment run was successful.

@adrikagupta adrikagupta force-pushed the fix-bq-source-partitioning branch 2 times, most recently from 1d9e162 to 21de8fc Compare September 24, 2025 13:55
@adrikagupta adrikagupta marked this pull request as ready for review September 24, 2025 14:09
@adrikagupta adrikagupta added the build Trigger unit test build label Sep 24, 2025
@adrikagupta adrikagupta force-pushed the fix-bq-source-partitioning branch from 21de8fc to 1d4aa48 Compare September 24, 2025 14:14
@adrikagupta adrikagupta force-pushed the fix-bq-source-partitioning branch from 1d4aa48 to 85633f7 Compare September 25, 2025 09:14
Copy link
Contributor

@itsankit-google itsankit-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one small comment, otherwise LGTM

@adrikagupta adrikagupta merged commit e00d81b into develop Sep 25, 2025
16 checks passed
@adrikagupta adrikagupta deleted the fix-bq-source-partitioning branch September 25, 2025 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants