Skip to content

bigquerydatatransfer: add google_bigquery_data_transfer_data_source_enrollment - #18564

Open
KyriosGN0 wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
KyriosGN0:bigquerydatatransfer-data-source-enrollment
Open

bigquerydatatransfer: add google_bigquery_data_transfer_data_source_enrollment#18564
KyriosGN0 wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
KyriosGN0:bigquerydatatransfer-data-source-enrollment

Conversation

@KyriosGN0

Copy link
Copy Markdown
Contributor

What

Adds google_bigquery_data_transfer_data_source_enrollment, a GA MMv1 resource wrapping projects:enrollDataSources / projects.locations:unenrollDataSources.

  • One data source per resource (data_source_id), following the google_project_service precedent noted in triage rather than the list shape suggested in the issue.
  • Enrollment is project-wide, so location is not part of the resource identity. It appears only as unenroll_location (default us) because the API has no project-level unenroll method and delete must route through some location; unenrolling through any one location removes the enrollment everywhere.
  • read_error_transform remaps the 400 FAILED_PRECONDITION returned for an un-enrolled data source to a 404, so deletion is detected as drift instead of erroring.

Why

Resolves hashicorp/terraform-provider-google#20217. Some data sources — Google Cloud Carbon Footprint exports in particular — must be enrolled before google_bigquery_data_transfer_config can target them, otherwise creation fails with Error 400: BigQuery DataTransfer is not enabled for <id>. That step is currently reachable only from the Cloud console, so a carbon export cannot be stood up from scratch in Terraform; users hand-click it or shell out to curl from a null_resource.

Note for reviewers: API eventual consistency

Three pieces of custom code exist solely to work around this, and I would rather flag it than have it read as over-engineering. After an enroll or unenroll, dataSources.get and dataSources.list both flap — consecutive reads alternate between success and FAILED_PRECONDITION depending on which replica serves them, for seconds to minutes. There is no consistency token.

  • post_create waits for several consecutive successful reads. Without it, the read following create hits a flapped 400 and Terraform fails with Provider produced inconsistent result after apply: Root object was present, but now absent. A single successful poll is not enough — in one reproduction the poll saw 200 and the next read 250ms later saw 400.
  • test_check_destroy polls for consecutive gone-reads, because the same flapping in reverse made the generated check report still exists after a successful destroy. Only the not-enrolled signal counts as gone, so auth or network errors are not mistaken for success.
  • exclude_import_test: true on the sample step. Importing a settled enrollment is reliable (20/20 reads on long-established enrollments); importing one created a second earlier is not, and no real user does that. post_create cannot help, since import performs its own read outside create. Create, read and destroy coverage is retained.

The window scales with recent churn: an untouched enrollment read 20/20, while one cycled ~25 times in 40 minutes dropped to ~6/20. A single enroll normally converges in a few seconds. This may be worth raising with the BigQuery Data Transfer team.

Testing

Verified against a real project: make provider VERSION=ga, go build ./..., go vet ./google/services/bigquerydatatransfer/..., mmv1 unit tests, and the generated acceptance test all pass. Also exercised manually through the Terraform CLI via dev_overrides: apply, clean follow-up plan, then an out-of-band unenroll correctly planning a recreate.

`google_bigquery_data_transfer_data_source_enrollment`

…nrollment

Enrolls a BigQuery Data Transfer Service data source in a project via
projects:enrollDataSources, so data sources that require enrollment (such as
Google Cloud Carbon Footprint exports) can be managed in Terraform instead of
being enabled by hand in the Cloud console.

Enrollment is project-wide, so location is not part of the resource identity.
It appears only as unenroll_location, because the API offers no project-level
unenroll method and delete must route through some location.

Reads of an un-enrolled data source return 400 FAILED_PRECONDITION rather than
404, so a read_error_transform remaps the code for drift detection. Enrollment
is also eventually consistent: reads alternate between success and
FAILED_PRECONDITION for a while after a write, which requires a post_create
wait and a polling destroy check. Importing a freshly created enrollment is
unreliable for the same reason, so the generated import test step is excluded.

Fixes hashicorp/terraform-provider-google#20217

Signed-off-by: AvivGuiser <avivguiser@gmail.com>
@modular-magician modular-magician added the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Aug 7, 2026
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@malhotrasagar2212, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting-approval Pull requests that need reviewer's approval to run presubmit tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing resource BigQuery DataTransfert projects.enrollDataSources

2 participants