Clarified some steps in fast/stages/0-org-setup to make it easier for newcomers - #4049
Conversation
juliocc
left a comment
There was a problem hiding this comment.
Thanks for the PR. Overall this looks good and it does improve the experience for newcomers.
I left a few comments below.
| ##### Provider Configuration for edge cases | ||
|
|
||
| In some cases, i.e: if you are running the bootstrap process from a Google Cloud Compute Engine (GCE VM), rather than your local laptop, the setup behaves differently. The issue is that Terraform detects it is running inside a GCE VM instance and falls back to using the GCE VM management project to track API quota usage. The workaround is to create a temporary `providers.tf` file in the `fast/stages/0-org-setup/` directory to explicitly declare the billing project to be used during bootstrap. | ||
|
|
||
| ```terraform | ||
| # Temporary providers.tf file to resolve quota project issues during bootstrap | ||
| provider "google" { | ||
| user_project_override = true | ||
| billing_project = "[project id]" | ||
| } | ||
|
|
||
| provider "google-beta" { | ||
| user_project_override = true | ||
| billing_project = "[project id]" | ||
| } | ||
| ``` | ||
|
|
There was a problem hiding this comment.
The way we usually manage this is by setting the default (or quota) project via gcloud.
I wouldn't suggest creating a provider file this way as it can cause issues later with the one that we generate/link later.
There was a problem hiding this comment.
I added this section because Terraform was ignoring the quota project I set via gcloud. Now, the machine I am running all this is a GCE VM (from a separate organization btw) and terraform was simply assuming the GCE VM's project as the quota project.
Do you know of another way to specify the quota project? Or do you think my use case is so niche that it might not be worth adding to the README.md?
There was a problem hiding this comment.
My my concern is that you might end up with multiple provider files.
@ludoo wdyt?
There was a problem hiding this comment.
+1 on not creating provider files, we configure the default project which is easier and works if left in place for subsequent runs
| To create `iam:` part of the `/organization/.config.yaml` file, you can use following snippet: | ||
| ##### IAM by Role (Authoritative) | ||
|
|
||
| The `iam:` block of the factory YAML file (`/organization/.config.yaml`) is authoritative, meaning that it will overwrite any existing IAM bindings on the resource, removing any binding for any role that is not explicitly listed. It is grouped by role name and contains a list of members. |
There was a problem hiding this comment.
meaning that it will overwrite any existing IAM bindings on the resource, removing any binding for any role that is not explicitly listed
This (seems) incorrect. Perhaps leave the introductory text but don't explain what an authoritative binding is
There was a problem hiding this comment.
Can you confirm if these three use cases correspond to the Terraform resources linked here:
IAM by Role (Authoritative)
iam:block corresponds to the Terraform resource https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_iam#google_project_iam_binding-1
IAM by Principal (Authoritative)
iam_by_principals:block corresponds to the Terraform resource https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_iam#google_project_iam_policy-1
IAM by Principal Additive (Non-Authoritative)
iam_by_principals_additive:block corresponds to the Terraform resource https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_iam#google_project_iam_member-1
There was a problem hiding this comment.
No, not exactly. iam and iam_by_principals is authoritative (uses *_iam_binding) while iam_by_principals_additive is non-authoritative (uses *_iam_member)
There was a problem hiding this comment.
authoritative for the roles it includes, it's not a IAM policy so it's not authoritative for the resource, and I agree with Julio this is not the place to explain it
| To create `iam_by_principals:` part of the factory YAML file, you can use following snippet: | ||
| ##### IAM by Principal (Authoritative) | ||
|
|
||
| The `iam_by_principals:` block of the factory YAML file (`/organization/.config.yaml`) is authoritative, meaning that it will overwrite any existing IAM bindings on the resource, removing any binfing for any principal that is not explicitly listed. It behaves exactly like `iam:` and is provided as a convenience for those who prefer to group IAM bindings by principal. |
ludoo
left a comment
There was a problem hiding this comment.
Thanks for the PR, some good bits in here. Left a few comments on some critical edits.
TBH the skill we have should be the default ingress path for users, and when run can be prompted to answer many of the questions addressed here which IMHO are too low level for this specific doc (how our IAM interface works for example).
| # - no admin access to billing account (don't set roles here or in org) | ||
| # Choose one of the following three modes of managing billing IAM: | ||
| # 1. at the org level: don't set the roles here. Delete the | ||
| # "iam_bindings_additive" block below and set roles in the organization |
There was a problem hiding this comment.
or simply delete the file, or change path for billing in factories_config (which preserves the file but ignores it)
| # "iam_bindings_additive" block below | ||
| # 3. no admin access to billing account: if your billing account is either | ||
| # shared or externally managed, don't set roles here nor in org. Delete the | ||
| # "iam_bindings_additive" block below and don't set roles in the organization either. |
| - one organization | ||
| - credentials with admin access to the organization and one billing account | ||
| - credentials with admin access to the organization | ||
| - one billing account |
| The admin principal is typically a group that includes the user running the first apply, but any kind of principal is supported. More principals (network admins, security admins, etc.) are present in some of the [default factories datasets](#default-factory-datasets), and others can be added if needed by editing the YAML configuration files. | ||
| You have sufficient permissions to manage billing IAM on either: | ||
| - an Organization-associated / Organization-level Billing Account: owned and managed directly within your Google Cloud Organization | ||
| - a Cross-Organization Billing or Reseller Billing Subaccount: externally managed billing account |
There was a problem hiding this comment.
with reseller accounts you almost never manage them, you only use them
| You have sufficient permissions to manage billing IAM on either: | ||
| - an Organization-associated / Organization-level Billing Account: owned and managed directly within your Google Cloud Organization | ||
| - a Cross-Organization Billing or Reseller Billing Subaccount: externally managed billing account | ||
| Get familiar with the [IAM relationship between organizations, projects and Cloud Billing accounts](https://docs.cloud.google.com/billing/docs/how-to/billing-access#relationships-between-resources) and refer to the [billing section](#billing-account-iam) for more details or non-standard configurations. |
There was a problem hiding this comment.
this is too much detail to give here, and not needed imho
|
|
||
| If you do not have sufficient permissions to manage billing IAM (common in "brownfield" environments or when using External / Reseller Billing Subaccounts), the IAM bindings need to be assigned via an external flow. Therefore, you need to remove billing IAM bindings from the YAML configuration files: | ||
| 1. remove billing IAM bindings (`iam_bindings_additive:`) from file `datasets/[dataset_name]/billing-accounts/default.yaml` and | ||
| 2. remove billing IAM bindings (`iam: > roles/billing.creator:`) from file `datasets/[dataset_name]/organization/.config.yaml` |
There was a problem hiding this comment.
harmless if they stay, so optional
| ##### If you cannot manage billing IAM | ||
|
|
||
| If you do not have sufficient permissions to manage billing IAM (common in "brownfield" environments or when using External / Reseller Billing Subaccounts), the IAM bindings need to be assigned via an external flow. Therefore, you need to remove billing IAM bindings from the YAML configuration files: | ||
| 1. remove billing IAM bindings (`iam_bindings_additive:`) from file `datasets/[dataset_name]/billing-accounts/default.yaml` and |
There was a problem hiding this comment.
just remove the file or redirect the factory to a dummy path, easier and cleaner
| ##### Provider Configuration for edge cases | ||
|
|
||
| In some cases, i.e: if you are running the bootstrap process from a Google Cloud Compute Engine (GCE VM), rather than your local laptop, the setup behaves differently. The issue is that Terraform detects it is running inside a GCE VM instance and falls back to using the GCE VM management project to track API quota usage. The workaround is to create a temporary `providers.tf` file in the `fast/stages/0-org-setup/` directory to explicitly declare the billing project to be used during bootstrap. | ||
|
|
||
| ```terraform | ||
| # Temporary providers.tf file to resolve quota project issues during bootstrap | ||
| provider "google" { | ||
| user_project_override = true | ||
| billing_project = "[project id]" | ||
| } | ||
|
|
||
| provider "google-beta" { | ||
| user_project_override = true | ||
| billing_project = "[project id]" | ||
| } | ||
| ``` | ||
|
|
There was a problem hiding this comment.
+1 on not creating provider files, we configure the default project which is easier and works if left in place for subsequent runs
| #### Importing org policies | ||
|
|
||
| If your dataset includes org policies which are already set in the organization, the first apply will fail with a `409 Conflict` error. In this case, you must either comment them out in the relevant YAML files or configure this stage to import them. To figure out which policies are set, run `gcloud org-policies list --organization [your org id]`, then set the `org_policies_imports` variable in your tfvars file. The following is an example. | ||
| If your dataset includes org policies which are already set in the organization, the first apply will fail with a `409 Conflict` error. In this case, you must either comment them out in the relevant factory YAML files (`fast/stages/0-org-setup/datasets/<YOUR_CHOSEN_DATASET>/organization/org-policies/*.yaml`) or configure this stage to import them. |
There was a problem hiding this comment.
we should link to the instructions for importing from here
| To create `iam:` part of the `/organization/.config.yaml` file, you can use following snippet: | ||
| ##### IAM by Role (Authoritative) | ||
|
|
||
| The `iam:` block of the factory YAML file (`/organization/.config.yaml`) is authoritative, meaning that it will overwrite any existing IAM bindings on the resource, removing any binding for any role that is not explicitly listed. It is grouped by role name and contains a list of members. |
There was a problem hiding this comment.
authoritative for the roles it includes, it's not a IAM policy so it's not authoritative for the resource, and I agree with Julio this is not the place to explain it
|
I'm closing this as it got no traction for the last 3 weeks. Feel free to reopen once comments have been accounted for. |
While going through Fast Fabric deployment, I've clarified a few parts of the README.md documentation. I hope this is helpful