bigquerydatatransfer: add google_bigquery_data_transfer_data_source_enrollment - #18564
Open
KyriosGN0 wants to merge 1 commit into
Open
Conversation
…nrollment Enrolls a BigQuery Data Transfer Service data source in a project via projects:enrollDataSources, so data sources that require enrollment (such as Google Cloud Carbon Footprint exports) can be managed in Terraform instead of being enabled by hand in the Cloud console. Enrollment is project-wide, so location is not part of the resource identity. It appears only as unenroll_location, because the API offers no project-level unenroll method and delete must route through some location. Reads of an un-enrolled data source return 400 FAILED_PRECONDITION rather than 404, so a read_error_transform remaps the code for drift detection. Enrollment is also eventually consistent: reads alternate between success and FAILED_PRECONDITION for a while after a write, which requires a post_create wait and a polling destroy check. Importing a freshly created enrollment is unreliable for the same reason, so the generated import test step is excluded. Fixes hashicorp/terraform-provider-google#20217 Signed-off-by: AvivGuiser <avivguiser@gmail.com>
|
Googlers: For automatic test runs see go/terraform-auto-test-runs. @malhotrasagar2212, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look. You can help make sure that review is quick by doing a self-review and by running impacted tests locally. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
google_bigquery_data_transfer_data_source_enrollment, a GA MMv1 resource wrappingprojects:enrollDataSources/projects.locations:unenrollDataSources.data_source_id), following thegoogle_project_serviceprecedent noted in triage rather than the list shape suggested in the issue.locationis not part of the resource identity. It appears only asunenroll_location(defaultus) because the API has no project-level unenroll method and delete must route through some location; unenrolling through any one location removes the enrollment everywhere.read_error_transformremaps the400 FAILED_PRECONDITIONreturned for an un-enrolled data source to a 404, so deletion is detected as drift instead of erroring.Why
Resolves hashicorp/terraform-provider-google#20217. Some data sources — Google Cloud Carbon Footprint exports in particular — must be enrolled before
google_bigquery_data_transfer_configcan target them, otherwise creation fails withError 400: BigQuery DataTransfer is not enabled for <id>. That step is currently reachable only from the Cloud console, so a carbon export cannot be stood up from scratch in Terraform; users hand-click it or shell out tocurlfrom anull_resource.Note for reviewers: API eventual consistency
Three pieces of custom code exist solely to work around this, and I would rather flag it than have it read as over-engineering. After an enroll or unenroll,
dataSources.getanddataSources.listboth flap — consecutive reads alternate between success andFAILED_PRECONDITIONdepending on which replica serves them, for seconds to minutes. There is no consistency token.post_createwaits for several consecutive successful reads. Without it, the read following create hits a flapped 400 and Terraform fails withProvider produced inconsistent result after apply: Root object was present, but now absent. A single successful poll is not enough — in one reproduction the poll saw 200 and the next read 250ms later saw 400.test_check_destroypolls for consecutive gone-reads, because the same flapping in reverse made the generated check reportstill existsafter a successful destroy. Only the not-enrolled signal counts as gone, so auth or network errors are not mistaken for success.exclude_import_test: trueon the sample step. Importing a settled enrollment is reliable (20/20 reads on long-established enrollments); importing one created a second earlier is not, and no real user does that.post_createcannot help, since import performs its own read outside create. Create, read and destroy coverage is retained.The window scales with recent churn: an untouched enrollment read 20/20, while one cycled ~25 times in 40 minutes dropped to ~6/20. A single enroll normally converges in a few seconds. This may be worth raising with the BigQuery Data Transfer team.
Testing
Verified against a real project:
make provider VERSION=ga,go build ./...,go vet ./google/services/bigquerydatatransfer/..., mmv1 unit tests, and the generated acceptance test all pass. Also exercised manually through the Terraform CLI viadev_overrides: apply, clean follow-up plan, then an out-of-band unenroll correctly planning a recreate.