GCP infrastructure support - #73
Closed
PoslavskySV wants to merge 39 commits into
Closed
Conversation
…hook The pre-install Job mounted the workspace PVC to run chown, but the PVC was rendered as a regular manifest — so on fresh install the hook fires before the PVC exists, leaving the Job stuck in Pending forever. Move the chown into a Deployment-level initContainer, conditional on the same workspace types (fsxLustre, filestore, nfs, pvc with chownOnCreate). The initContainer runs after the PVC is bound, so timing is correct by construction. Slightly increases every pod restart cost (chown -R is a no-op after first run), but eliminates the install deadlock. Delete hook-chown-workspace.yaml; the initContainer replaces it.
Spike implementation of the end-to-end GCP install matching the AWS
CloudFormation runbook's surface. Provisioned via `tofu apply` locally;
destined to be wrapped in Google Infrastructure Manager as the Tier 1
one-click flow.
What's here:
- APIs: enables container, file, dns, iam, secretmanager, certificatemanager,
servicenetworking, config (IM), storage, and related services.
- Network: VPC + subnet with VPC-native secondary ranges (pods/services)
and a PSA global address + service-networking peering so Filestore Zonal/
Regional/Enterprise tiers can mount via private service access.
- GKE Standard zonal cluster with VPC-native, Workload Identity, Gateway
API, and the Filestore + PD CSI drivers. Three node pools:
system (always-on n2d-standard-4 × 2)
ui (0-4 n2d-standard-4, tainted dedicated=ui)
batch (0-N n2d-highmem-64, tainted dedicated=batch)
- Storage: Filestore (default ZONAL SSD per CTO validation — HDD does not
handle real bioinformatics load), primary GCS bucket with CORS + uniform
access, random suffix to avoid naming collisions.
- IAM: two GCP service accounts (platforma-server, platforma-jobs), each
granted roles/storage.objectAdmin on the bucket and roles/iam.workload
IdentityUser for the matching K8s SA. Both also granted roles/iam.service
AccountTokenCreator on themselves so Platforma can presign GCS URLs for
the Desktop App (signBlob isn't in the default WI scope).
- Controllers: Kueue Helm release (matching AWS values-aws file's feature
gates + integrations) and AppWrapper install.yaml applied via the
kubectl provider.
- Platforma Helm release: points at the local chart (..charts/platforma)
for the spike; production will pull from ghcr.io. Passes cluster-derived
values for Filestore, GCS, service-account emails, license secret,
auto-generated htpasswd admin credentials (stored in Secret Manager),
Kueue queue quotas, and node selectors/tolerations matching the pool
labels.
Known gaps (follow-up work):
- No ingress / TLS yet — server validated via kubectl port-forward. Add
static IP + Cloud DNS A record + Certificate Manager cert + Gateway
when productizing.
- Infrastructure Manager metadata.yaml not yet written.
- No pre-flight quota check — fresh GCP projects will fail on N2D_CPUS,
Enterprise Storage (for SSD Filestore), and CPU-all-regions global.
Will add as a required resource before node pool creation.
- No quota_preference resources — need to bundle for auto-request.
- Local state file uses a hardcoded GCS backend bucket
(platforma-gcp-test-tfstate); IM users will need their own.
…libraries Phase 1+2 work on top of the initial GCP Terraform module commit: * Deployment size presets (small/medium/large/xlarge) — single var.deployment_size drives node pool maxes, Kueue queue quotas, Filestore default capacity, and quota-request amounts. Per-job cap fixed at 62 vCPU / 500 GiB to match AWS CloudFormation parallelism. * HTTPS ingress — gated on var.ingress_enabled. Provisions a global static IP, Cloud DNS A record, Certificate Manager managed cert with DNS authorization, GKE Gateway (Gateway API) bound to the cert via certmap annotation, and an HTTPRoute. The Gateway listener omits the tls block so the CEL "certificateRefs or options must be specified when mode is Terminate" rule doesn't fire — cert comes from the certmap annotation alone. * gRPC backend — Platforma server speaks gRPC on port 6345 (HTTP/2 only). The chart's Service has no appProtocol set so GKE Gateway defaults to HTTP/1.1 and marks the backend unhealthy. We provision a parallel Service "platforma-grpc" with appProtocol=kubernetes.io/h2c plus a HealthCheckPolicy doing GRPC health checks on the serving port. HTTPRoute targets the parallel Service. (Will be replaced once chart PR #71 lands and we set app.service.appProtocols.grpc on the chart's own Service.) * Quota auto-request — google_cloud_quotas_quota_preference resources for CPUS_ALL_REGIONS (global), N2D_CPUS / SSD / INSTANCES (per region), and Filestore Zonal storage. Skip-list var (skip_quota_requests) for projects that already have user-managed preferences for the same (service, quota_id, dimensions) — Cloud Quotas API rejects duplicate creates and exposes no DELETE. * Data libraries — var.data_libraries accepts a mixed list of GCS-typed (Workload Identity for same-project, no creds) and S3-typed (cross-account access keys) read-only buckets. Demo library toggle (var.enable_demo_data_library, default true) mounts MiLaboratories' AWS S3 demo bucket cross-cloud, mirroring the AWS CloudFormation default. * Outputs — pragmatic set covering both basic users (platforma_url, default_username, password_secret_console_url, post_deploy_steps) and operators (kubectl_credentials_command, port_forward_command, cluster_name, region, gcs_bucket). * Provider user_project_override + billing_project — required for the Cloud Quotas API when authenticating via local Application Default Credentials. * deploy_platforma toggle — count-gated helm_release.platforma so infra + cluster controllers (Kueue, AppWrapper) can be installed without the application for isolated infra testing. * platforma_image_override — passes a custom image:tag to the chart. * Cluster sized n2d-highmem-64 batch nodes (62 vCPU / ~500 GiB allocatable per node) so a single 50 vCPU / 400 GiB job fits.
… for IM Phase 3 prep. Infrastructure Manager's managed Terraform runner manages state internally and rejects bundles that include their own backend block. The block stays on disk for local-development workflows; the CI packaging step (Phase 4) excludes backend.tf when zipping the source for upload to gs://platforma-infrastructure-manager/. Also lower required_version from >= 1.9 to >= 1.5 — IM runs Terraform 1.5.7 and the higher floor would block the IM path. Local OpenTofu 1.11+ and Terraform 1.5+ both still work.
Triggers on `gcp-im-v*` tags (or manual dispatch). Stages the Terraform module, strips local-only files (backend.tf, terraform.tfvars, .terraform/), bundles the chart at terraform/platforma/ for the path helm_release expects in the IM context, rewrites the chart reference in app.tf accordingly, and uploads the tarball to gs://platforma-infrastructure-manager/. On a tagged release, also updates platforma-gcp-latest.tar.gz so the Cloud Shell tutorial (Phase 5) can default to the latest stable version. Auth: uses GCP_PUBLISHER_SA_KEY repo secret (service-account JSON key) for now. WIF (workload identity federation) upgrade is a follow-up.
Adds the Tier-1 user path from the GCP runbook plan: a one-click "Open in
Cloud Shell" experience that walks a non-IT user through the few inputs
needed (deployment name, region, size, domain, DNS zone, license key,
contact email) and submits a Google Cloud Infrastructure Manager
deployment pointing at the published TF bundle in
gs://platforma-infrastructure-manager/.
Files:
- tutorial.md Cloud Shell walkthrough rendered in the side panel
- install.sh driver script (preflight checks, prompts/env vars,
service account creation, IM apply, poll, outputs)
- README.md usage notes including the Open-in-Cloud-Shell URL and
instructions for bumping IM_BUNDLE_VERSION on each release
The bundle URL in install.sh is intentionally versioned (mirroring the
AWS CloudFormation runbook's versioned-template-URL approach). Tutorial
will be updated on each release by bumping IM_BUNDLE_VERSION and merging
to main; users always pick up the latest tutorial code on their next
Cloud Shell click since Cloud Shell clones HEAD.
Phase 5 follow-up — adds the missing auth and data-library inputs that
the install path needs for real production use. Without these, a user
can only test with the bundled demo data and an auto-generated random
password (testing only).
Terraform module:
* New auth_method variable (htpasswd|ldap; default htpasswd).
* New htpasswd_content variable for user-supplied bcrypted content.
Empty (default) keeps the auto-gen path; non-empty creates a
kubernetes_secret 'platforma-htpasswd-provided' that the chart
references via auth.htpasswd.secretName.
* New ldap_* variables — ldap_server, ldap_start_tls, ldap_bind_dn
(direct-bind), ldap_search_rules / ldap_search_user /
ldap_search_password (search-bind). Search password lands in a
kubernetes_secret referenced via auth.ldap.searchPasswordSecretRef
(chart's standard pattern).
* app.tf: refactored auth = {...} block to a locals.auth_helm_value
that branches on auth_method + htpasswd_content presence. Added
conditional kubernetes_secret resources (count = 0 when not used).
* terraform.tfvars.example documents all new fields plus the
production-vs-testing tradeoff for htpasswd auto-gen.
Cloud Shell installer (install.sh):
* New 'Authentication' section. Default htpasswd path is silent (warn
about testing-only). LDAP path prompts for the 6 LDAP fields. Power
users skip prompts entirely by exporting HTPASSWD_CONTENT before
running.
* New 'Data libraries' section. Prompts for ENABLE_DEMO + an integer
count of additional libraries (0-3), then loops once per library
collecting name/type/bucket/prefix/region/keys. Power users skip
prompts by exporting DATA_LIBRARIES_YAML with arbitrary YAML.
* LDAP search rules use ; separator (commas appear inside DNs).
* Switched IM submission from --input-values=k=v,… to YAML
--input-values-file=… because data_libraries / ldap_search_rules
are list/object types not expressible in the comma form.
* Added a YAML pretty-print of the inputs file (with secrets
redacted) before submission so the user can sanity-check.
install.sh queries the Cloud Quotas API at startup, finds any
user-managed QuotaPreferences that would collide with our auto-request
resources (Cloud Quotas API rejects duplicate creates and exposes no
DELETE), and:
1. Auto-populates skip_quota_requests in the inputs file so the
deployment doesn't fail with a confusing 409.
2. Compares each existing preferred_value against what the chosen
deployment_size needs and warns when it's below — the deployment
would still create successfully but jobs may fail to scale up
later. User can confirm continue or abort.
Lookup tables for required values mirror presets.tf — keep in sync.
Surfaces the deployment_size preset values (parallel jobs, UI nodes,
Filestore size, all 5 quota amounts) in three places:
- infrastructure/gcp/cloudshell/tutorial.md — short version users
see during install (just parallel-jobs / UI / Filestore, plus a
note about quota auto-request and human-review thresholds).
- infrastructure/gcp/cloudshell/README.md — internal/dev-facing
table including the keep-in-sync notes for install.sh's lookup
tables and the quota collision handling logic.
- infrastructure/gcp/README.md — new stub runbook (full docs come
in Phase 7) with the user-facing preset table, three install
tiers (Cloud Shell / local CLI / local Terraform), auth options
(htpasswd auto/content / LDAP), data library options.
Source of truth remains presets.tf — comments in install.sh and the
internal README call out the keep-in-sync requirement.
…NAME Previous prompts asked users to type DOMAIN_NAME and DNS_ZONE_NAME free-form, which is error-prone — a non-IT user might type any nice URL without realizing it must be a subdomain of a Cloud DNS managed zone they own. Now the installer queries gcloud dns managed-zones list, presents a numbered picker (auto-selects when there's only one zone), and prompts only for a SUBDOMAIN PREFIX. The full FQDN is derived as <prefix>.<zone-dns-name>. Errors out with actionable guidance when the project has zero zones (pointing at the create command and domain-guide.md). Both DOMAIN_NAME and DNS_ZONE_NAME env vars still bypass the picker for non-interactive runs.
Replace the hidden license key prompt with a visible one (so users can
spot paste typos) and validate the entered key against
licensing-api.milaboratories.com/refresh-token before proceeding.
Same endpoint the Platforma server hits at runtime — a key that passes
here is effectively certain to start the cluster successfully.
Behaviour:
- 200 from licensing API: accept and continue
- 4xx (e.g. 403 unknown key): print HTTP code + extract any
.message / .error / .errors field from the response body, re-prompt
- Network failure: shows curl exit and re-prompts (user can Ctrl-C)
- LICENSE_KEY env var: validated up-front; if it fails, fall through
to interactive prompt rather than failing the whole install
Override the endpoint with LICENSING_API_URL for staging tests.
…ty default"
Previously prompt_var rejected empty input whenever the default was the
empty string, which misfired on three legitimately-optional prompts:
- data-library prefix
- data-library access_key (empty = use Workload Identity / IAM role)
- LDAP bind DN (empty = use search-bind path)
Bash can't distinguish "no 3rd argument" from "3rd argument is empty
string" via "${3:-}" alone. Use the arg count instead: passing an
empty default explicitly marks the prompt as optional and lets empty
input through.
The previous attempt used --input-values-file=… which doesn't actually exist in gcloud. gcloud's --input-values flag is comma-separated KEY=VALUE pairs only — no way to express list/object types like data_libraries or ldap_search_rules. Pivot: download the published bundle locally, write user inputs as inputs.auto.tfvars.json (Terraform auto-loads any *.auto.tfvars.json), submit via --local-source. Same end result, no flag-shape limitations. Other fixes: * Replaced YAML output with JSON (cleaner construction via jq) * Moved temp directory + EXIT trap to the top of the script so per-function locals don't go out of scope before cleanup runs (was triggering "inputs_file: unbound variable" under set -u)
On a fresh GCP project, the first 'gcloud infra-manager deployments apply'
fails with:
Creating the root Cloud Storage bucket failed: Not found;
Spanner: Not found ServiceAccountInfoDataType
…because the Infrastructure Manager service identity
(service-<project_num>@gcp-sa-config.iam.gserviceaccount.com) is not
provisioned automatically when the API is enabled — only when something
explicitly requests it. Subsequent calls then work because the identity
gets created on first failure, but the failure is the apply itself.
Fix: call 'gcloud beta services identity create --service=config.googleapis.com'
in preflight, right after enabling the API. Returns immediately with the
SA email and provisions the underlying Spanner record. Fully idempotent.
Also passed shellcheck (-S warning) and bash -n.
Two bugs that OpenTofu 1.11 (used locally) silently accepted but TF 1.5.7
(used by Infrastructure Manager) rejects at plan time:
1. auth_helm_value conditional type mismatch
The htpasswd-content branch produced { secretName, secretKey } while
the htpasswd-auto branch produced { credentials = [...] }. TF 1.5
strictly unifies conditional branch types and rejects with:
"Inconsistent conditional result types ... Type mismatch for object
attribute htpasswd: The 'true' value includes object attribute
secretKey, which is absent in the 'false' value."
Refactor: split the three auth shapes into named locals and merge()
mutually-exclusive single-key objects with empty fallbacks. merge
doesn't trigger the unify check because each branch evaluates
independently.
2. workspace_capacity_gb validation short-circuit
'var.workspace_capacity_gb == null || var.workspace_capacity_gb >= 1024'
evaluates BOTH operands in TF 1.5 (no short-circuit inside validation
blocks), and 'null >= 1024' errors with "argument must not be null"
before we ever check for null. Replace || with explicit ternary.
Validated end-to-end: deployed to a real fresh project with these
inputs; reached state=ACTIVE with cluster RUNNING and Filestore READY.
If a previous IM apply failed before any revision was created (e.g. because the IM service identity wasn't yet provisioned, the configured service account lacked permissions, or — historically — the TF source had a plan-time error and never reached r-0), gcloud's apply command crashes with 'IndexError: list index out of range' when trying to read latestRevision on the next apply attempt. Detect this state explicitly: when describe returns state=FAILED|ACTIVE with empty latestRevision, delete the deployment first, then create fresh.
… checkout
Source-of-truth simplification. Previously install.sh downloaded a published
tarball from gs://platforma-infrastructure-manager/, which required a CI
workflow, a service-account JSON key as a GitHub secret, a public bucket,
and a separate IM_BUNDLE_VERSION constant kept in lockstep with releases.
The bundle the user actually needs is whatever's in their git checkout —
Cloud Shell already clones the repo, and standalone runners need the repo
anyway. install.sh now assembles the IM source bundle in-process from
sibling directories (../terraform + ../../charts/platforma), applies the
same transformations the workflow used to do (drop backend.tf, bundle the
chart, rewrite the chart path in app.tf), and submits via --local-source.
Versioning + updates simplify accordingly:
- default Cloud Shell button → main → latest
- pinned production releases → cloudshell_git_branch=gcp-im-v0.1.0 in
button URL → that tag
- updates: re-open Cloud Shell (re-clones HEAD), re-run install.sh,
IM creates a new revision against the new TF source
What's gone:
- .github/workflows/publish-gcp-im.yaml
- GCS bucket dependency (gs://platforma-infrastructure-manager — the
bucket itself can be deleted manually; nothing references it now)
- GCP_PUBLISHER_SA_KEY GitHub repo secret (no longer used)
- IM_BUNDLE_VERSION / IM_BUNDLE_URL constants in install.sh
Also (Q3): htpasswd content can now be supplied as a file path
- HTPASSWD_FILE env var (path), or
- interactive prompt asking for path (empty = auto-gen testing path)
- HTPASSWD_CONTENT env var still works (priority over HTPASSWD_FILE)
Validated: bundle assembly produces a tofu-valid module on the local
checkout (head of gcp-infrastructure branch).
The 'Proceed? [y/N]' confirmation at the end of input collection treated
empty Enter as 'no' and aborted, forcing the user to re-enter all 10+
prompts. Same behaviour on the 'Continue anyway?' confirmation when
existing quotas are below preset requirements.
Replace both call sites with a new prompt_yn helper that loops on empty
or invalid input ('Please answer y or n.') and only returns when an
explicit y or n is given. No default — accidental Enter no longer wipes
out a long input session.
…A resources Several resources in dns_tls.tf and network.tf had no depends_on on google_project_service.enabled, so on a fresh project they could race with API enablement and fail with: Error 403: Certificate Manager API has not been used in project ... before or it is disabled. (Hit on platforma-gcp-test-2.) Affected resources: - google_dns_record_set.platforma - google_dns_record_set.cert_validation - google_certificate_manager_dns_authorization.platforma - google_certificate_manager_certificate.platforma - google_certificate_manager_certificate_map.platforma - google_certificate_manager_certificate_map_entry.platforma - google_compute_global_address.private_service_access Other resources without explicit depends_on are fine — they reference parent resources (cluster, bucket, SA, etc.) that already wait for google_project_service.enabled, so the dependency holds transitively.
…ction Cloud Shell sessions disconnect after ~1 hour idle. The 20-25 min deployment poll loop in install.sh dies with the session, but the deployment itself continues in the background. Users need to know where to resume watching from a browser. install.sh: print Infrastructure Manager + Cloud Build console URLs immediately after submitting the deployment (before the long poll loop), so they're in the terminal scrollback even if the session later disconnects. tutorial.md: new "Monitoring + recovery" section pointing at the same two URLs, listing common failure modes (quota, cert validation), and documenting that re-running install.sh on an existing deployment is safe (it updates in place via a new IM revision).
…ced guides
Phase 7 of the GCP rollout. Four new/expanded docs covering everything
beyond the Cloud Shell tutorial:
* README.md promoted from stub to full runbook landing page —
architecture diagram, prerequisites, cost notes,
install paths overview, pricing, auth + data-library
reference, troubleshooting (cert validation,
quota errors, "no healthy upstream"), tearing down,
tracking issues for the open chart-side PRs.
* domain-guide.md step-by-step Cloud DNS zone setup + NS delegation
from external registrars: Route53, Cloudflare,
GoDaddy, Namecheap, generic. Verification via dig.
Common pitfalls (trailing dot, propagation, root
delegation). Multiple-deployment patterns.
* permissions.md 12-role fine-grained IAM set replacing roles/owner
on the deployer service account. Per-role
justification mapped to the resources the TF module
creates. Apply-script + cross-project DNS notes
+ org-policy interactions (DRS, VPC-SC).
* advanced-installation.md Tier-3 power-user path: local Terraform
without Infrastructure Manager. State backend
setup, gcloud auth, customization recipes (LDAP,
pre-bcrypted htpasswd, mixed GCS+S3 data libraries,
cross-project DNS, custom node-pool sizing,
skip_quota_requests), updates, destruction,
comparison to the IM path.
Step 2 (with-ingress) and step 3 (port-forward) of post_deploy_steps now branch on auth_method: - ldap → "Log in with your LDAP credentials (server: <ldap_server>)." - htpasswd + user-supplied content → "Log in with one of the user/password pairs from your htpasswd_content." - htpasswd + auto-generated → existing Secret Manager URL + CLI command Final step now reads "Add Connection → Remote Server, connect to <url>." without the "log in as ... with the password above" tail, since the credential location depends on auth method. The htpasswd_content emptiness check is wrapped in nonsensitive() because htpasswd_content is sensitive; without the wrap, the entire post_deploy_steps output inherits sensitivity and TF refuses to print it.
Catch the most common first-time-user failure mode early: a Cloud DNS zone exists, but the parent zone has not delegated NS authority to it. Without this check, install.sh runs all the way through provisioning and then Cert Manager sits in PROVISIONING / AUTHORIZATION_ISSUE indefinitely — costing the user 20+ minutes to discover the problem. verify_dns_delegation() runs after collect_inputs and before detect_existing_quota_prefs: - describes the chosen zone, gets dnsName + nameServers - runs `dig +short NS <dnsName>` from the user's machine - compares to the 4 ns-cloud-*.googledomains.com nameservers - on empty/mismatch: prints the exact NS records to add at the parent zone, points at domain-guide.md, asks "proceed anyway?" (allows debugging flows that aren't first-install) Skipped when INGRESS_ENABLED=false (no cert validation, no delegation needed) or when `dig` is unavailable (warns instead of failing). Tested locally against: - a valid delegation (gcp-test.platforma.bio in platforma-gcp-test-3) — passes - non-ingress mode — correctly skipped - a synthesized undelegated zone — fails with the expected guidance
The previous failure message said "add NS record at the parent zone" —
correct for a delegated subdomain, but misleading when the user owns the
root domain (e.g. bought through Cloud Domains, or registered at GoDaddy
and pointed nameservers at Cloud DNS). In that case there's no parent
zone they control; they need to update nameservers at the registrar.
New message lists both options:
- If <dnsName> is a domain you own at a registrar: update NS in the
registrar admin.
- If <dnsName> is a subdomain of a zone you already control: add an NS
record on the parent zone.
The detection logic itself was already correct for both cases — only the
remediation guidance needed widening.
Three-tier validation runs after DNS precheck, before quota / IM SA setup:
Tier 1 (syntax — hard fail):
- LDAP_SERVER parses as ldap://host[:port] or ldaps://host[:port]
- default port inferred (389 / 636) when absent
- StartTLS+ldaps mismatch warned
- exactly one bind mode set (direct-bind XOR search-bind)
- direct-bind: warn if LDAP_BIND_DN lacks '%u' placeholder
- search-bind: each rule matches '(filter)|baseDN', warn if no '%u'
Tier 2 (network — warn only):
- TCP probe via nc -z -w 5 (portable across BSD nc / Linux nc)
- TLS handshake via openssl s_client for ldaps:// — verify cert chain
- warn-only because this machine's reachability != GKE cluster's
reachability (Cloud Shell is behind GCP NAT, typically can't reach
corporate LDAP, but the cluster with VPN/peering can)
Tier 3 (bind — warn only, search-bind only):
- if ldapsearch is available and creds set, attempt anonymous-readable
search at the first rule's baseDN
- skipped for direct-bind (needs a real user's password to test)
Validated against:
- htpasswd auth method → no-op
- bad URL → fail with clear message
- missing bind mode → fail
- both modes set → fail
- malformed search rule → fail
- bind DN without %u → warn
- reachable TLS endpoint (google:443 standing in for ldaps) → TLS OK
- unreachable host → warn, continues
Cloud Shell ships without nc, so the previous nc-only probe printed "nc not found — skipping TCP reachability check" instead of doing the work. New chain tries python3 first (universally available on Cloud Shell and modern macOS, has clean socket timeout via socket.create_connection), falls back to nc (BSD or Linux flag form -z -w 5), and finally bash's /dev/tcp builtin (no clean timeout but fast-fails on most networks). The success line now reports which method ran: ✓ TCP host:port reachable from this machine (python3) Verified against ldap.eu.foxpass.com:636 with all three paths exercised (real run + simulated nc-missing → python3 fallback).
The previous two-sentence stub was uninformative for the most common question: "I want to add a data library / pull a new chart version / change auth — what do I actually do?" New section covers: - What you typically update (releases, libraries, auth, resize) and what data survives across updates (Filestore PVC, GCS bucket). - Path 1 — re-run install.sh with env-var pre-fill. Concrete copy-paste block listing every variable, with the data-libraries YAML caveat that the list is replaced as a whole (so include existing libraries when adding a new one). Notes that env-var pre-fill is a workaround until we read inputs back from the IM revision directly. - Path 2 — edit inputs in the IM Console. Quickest for one-off changes without setting up env vars. - Path 3 — Tier-3 local Terraform. Recommended for ongoing ops; one-line edit + tofu apply. Points at advanced-installation.md for migration. - Rollbacks via IM Console Revisions tab (Tier-1/2) or git revert + tofu apply (Tier-3).
…ckets
Previously the user had to run `gcloud storage buckets add-iam-policy-binding`
manually before listing a same-project GCS bucket in data_libraries. Skipping
the grant produced silent pod-init failures with "AccessDenied" — discovered
only when the pod couldn't initialise its data sources.
Two new google_storage_bucket_iam_member resources, one for platforma-server
and one for platforma-jobs, granting roles/storage.objectViewer on every
data library entry where:
- type == "gcs"
- project_id == "" (defaults to the deployment project) OR
project_id == var.project_id (explicit same-project)
Cross-project entries (project_id set to a different project) are skipped —
the deployer SA can't touch IAM on a foreign project without extra setup;
those still require pre-grant.
S3 entries are unaffected (auth via HMAC keys, not Workload Identity).
Updated README and advanced-installation.md to drop the "grant the role
BEFORE installing" instruction and clarify when manual grant is still needed
(cross-project only).
…em-64 The previous default (n2d-standard-16) was inconsistent with everything else in the module: kueue_max_job_cpu defaults to 62 vCPU, the small preset's n2d_cpus_quota requests 512 vCPU (sized for 4 × n2d-highmem-64), the README docs name "n2d-highmem-64 instance-hours" as the active-job cost, and the quota auto-request justification literally says "Platforma batch pool uses n2d-highmem-64 nodes". What that mismatch produced in practice: A typical Platforma job pod requests 16 vCPU + 64 GiB. After GKE daemonset/system overhead, an n2d-standard-16 node has only ~15.9 vCPU and ~57 GiB allocatable — pods short by 110m CPU and 7 GiB memory, sit Pending forever, autoscaler can't provision more (max reached or "CapacityIsNotFound"), and MiXCR clonotyping (or any real workload) is silently broken until the user diagnoses node-pool sizing. The new default has 64 vCPU / 512 GiB → ~62 vCPU / 500 GiB allocatable, matching kueue_max_job_cpu/memory exactly. One large job per node. Variable description now flags the trap explicitly: smaller types appear cheaper but require also lowering kueue_max_job_cpu/memory; otherwise pods sit Pending. Plus a note about N2D-CPUS-per-project-region quota implications. This is a default change — existing deployments need `tofu apply` (Tier-3) or re-running install.sh (Tier-1/2) to pick it up. The change recreates the batch node pool (machine_type is recreate-only).
…pools)
Previously the GCP module had a single batch node pool with one machine
type (n2d-highmem-64 after the recent default fix). Every job — whether
it asks for 16 vCPU or 60 vCPU — got a whole 64-vCPU / 512-GiB node, paying
for idle capacity proportional to the gap.
AWS already does this right: 5 batch node groups (m7i.4xlarge / .8xlarge /
.16xlarge + r7i.8xlarge / .16xlarge) sharing the same K8s label and taint.
The autoscaler picks the smallest-fitting group per pending pod. CTO
flagged that GCP wasn't matching this; this commit closes the gap.
Implementation:
- presets.tf grows a batch_pool_specs map with 5 shapes (16c-64g, 32c-128g,
64c-256g, 32c-256g, 64c-512g), each pinned to the matching n2d-* machine
type. Per-preset batch_pool_max_nodes mirror AWS's MaxBatch16c64g /
MaxBatch32c128g / etc. exactly so the same deployment_size label means
the same parallelism on both clouds.
- gke.tf becomes for_each over the spec map. Pool names are batch-${shape}.
All pools carry the same role=batch label, dedicated=batch:NoSchedule
taint, and an extra platforma.bio/batch-pool=<shape> label for diagnostics.
- google_container_cluster.primary now sets cluster_autoscaling
autoscaling_profile = OPTIMIZE_UTILIZATION — the GKE-side counterpart of
AWS's "least-waste" expander. Without it, the autoscaler picks pools at
random when multiple match a pending pod, defeating the multi-pool win.
- variables.tf drops batch_pool_machine_type and batch_pool_max_nodes
(single-pool concepts); adds batch_pool_max_nodes_overrides map keyed by
shape ID for per-pool overrides.
- Kueue ClusterQueue total CPU/memory now derives from
sum(pool_count × pool_capacity), so admission gate matches the cluster's
real upper bound rather than parallel_jobs × per-job-cap. Per-preset
totals (small=320c/1792Gi, medium=640c/3584Gi, large=1280c/7168Gi,
xlarge=2560c/14336Gi) validated via tofu console.
- quotas.tf justifications updated to reflect 5-pool layout.
- README adds per-pool max-node table per preset, pool-shape mapping with
AWS counterparts, and revised cost notes ($/hour per pool shape so users
can see the multi-pool win — a 16c MiXCR job on 16c-64g costs ~$0.90/h
vs ~$3.60/h on 64c-256g).
- advanced-installation.md replaces the old single-pool override snippet
with the new map-based syntax.
Chart unchanged: ResourceFlavor selects on role=batch and tolerates
dedicated=batch — both shared across all 5 pools, so a single flavor still
covers everything.
Fresh-project apply intermittently fails with:
Error 400: Identity Pool does not exist
when google_service_account_iam_member.{server,jobs}_wi tries to add a
WI binding for <project>.svc.id.goog before GKE has finished provisioning
the cluster's Identity Pool.
The pool is created lazily by GKE after the cluster comes up. Without an
explicit dep, Terraform parallelises and the IAM binding API call beats
the pool's existence — race window of seconds-to-minutes on a cold
project where everything provisions concurrently.
Fix: add depends_on = [google_container_cluster.primary] on both WI
bindings so TF serialises cluster-first, bindings-after.
Reproducible on platforma-gcp project on first apply; resolved by re-
applying after a few minutes (the pool exists by then). This commit
removes the need to retry.
Lightweight ADR log capturing the non-obvious choices made during the GCP module build. 21 entries covering: three-tier install model, deployment size presets matching AWS, per-job 62/500 cap, multi-pool batch design, Filestore Zonal SSD default, Workload Identity for runtime SAs, htpasswd auto-gen vs user-supplied, Cloud Quotas best-effort, --local-source bundle, bundled chart workaround, parallel grpc Service workaround, chown initContainer Filestore constraint, auto-grant SA on same-project GCS, IM as source of truth, custom HCL vs standard modules, deployer SA roles, gcp-im-v tag prefix, license validation, DNS delegation precheck, LDAP 3-tier precheck, demo data library cross-cloud. Each ADR follows a Decision / Why / Trade-offs / Revisit shape — small enough to scan, structured enough to cross-reference. Numbered for stable references; do not renumber existing entries. The "custom HCL vs standard modules" entry (ADR 15) addresses the question the CTO raised about maintainability: keep custom for v1.0, plan a network.tf migration to terraform-google-modules/network/google for v1.1 as a low-risk first step. Linked from the README's Files section so future maintainers find it before making structural changes.
Three changes that fix the same root cause: with 5 batch pool shapes plus
UI + system pools, peak node count at xlarge is ~30. Each public-IP node
consumed 1 IN_USE_ADDRESSES slot — far above the GCP default of 8.
Fresh-project installs hit the cap during scale-up with confusing Cluster
Autoscaler errors.
1. Private nodes (gke.tf)
private_cluster_config now sets enable_private_nodes = true,
enable_private_endpoint = false. Worker nodes have no external IPs;
control plane stays public so kubectl from any machine still works
without a bastion or IAP tunnel. master_ipv4_cidr_block defaults to
10.10.0.0/28 (configurable via variable).
2. Cloud NAT for egress (network.tf)
New google_compute_router + google_compute_router_nat. AUTO_ONLY IP
allocation (Google manages NAT IPs, typically 1-2 per region with
dynamic port allocation). Provides egress for nodes pulling container
images, reaching cross-cloud demo data (S3), external LDAP, and the
licensing API. Google APIs (Storage, IAM, Logging, Monitoring) bypass
NAT via Private Google Access on the nodes subnet — faster and avoids
NAT processing fees on the high-volume GCS traffic that bioinformatics
workloads generate.
Cluster depends_on the NAT so the first system-pool nodes don't hang
trying to pull from gcr.io with no egress path.
3. IN_USE_ADDRESSES auto-request (quotas.tf, presets.tf)
With private nodes + NAT, IN_USE_ADDRESSES usage drops to ~2-3 per
region (NAT IPs + GKE Gateway static IP). Default 8 would suffice but
we request modest headroom: 16/16/24/32 across small/medium/large/
xlarge. Without this preset entry, the previous public-nodes design
was silently broken for any preset above small on a fresh project.
Cost trade-off:
- Cloud NAT base ~$0.045/h + $0.045/GB egress processed.
- GCS traffic bypasses NAT (Private Google Access), so per-GB cost is
bounded to image pulls + occasional non-Google egress — typically
pennies per day for a small deployment.
Other updates:
- architecture-decisions.md ADR 22: documents the design + revisit
triggers (static egress IPs if customers whitelist us; per-cluster NAT
primitives if GCP ships one).
- README architecture diagram redrawn to show NAT + PGA flows.
- README deployment-size table adds the IN_USE_ADDRESSES column.
- permissions.md: clarifies compute.networkAdmin now also covers Cloud
Router + Cloud NAT.
For existing deployments updating to this commit: the cluster recreates
node pools (private_nodes is on cluster.private_cluster_config, which is
recreate-only in TF). Plan output will show this; not in-place. After
recreate, jobs resume on private nodes with NAT egress.
The previous commit (43908a1) made private nodes unconditional. That works cleanly for fresh installs but breaks the upgrade path for existing public-nodes deployments — private_cluster_config.enable_private_nodes is ForceNew, so flipping it would force cluster recreation, and TF can't plan kubernetes_namespace / helm_release / kubectl_manifest resources when their underlying cluster is being replaced (the kubernetes provider falls back to localhost:80 because google_container_cluster.endpoint is unknown during plan). Add var.enable_private_nodes (default true). New installs get private nodes + Cloud NAT as designed. Existing public-nodes deployments override to false on their next apply, keeping cluster intact; they migrate to private nodes via a deliberate destroy+recreate when convenient. Both private_cluster_config (gke.tf) and the Cloud Router + NAT (network.tf) are now gated on the variable. Cluster's depends_on still references google_compute_router_nat.primary — TF resolves the empty case (count = 0) as a no-op dependency. Migration notes: - Existing deployments upgrading to this commit must set enable_private_nodes = false in their inputs to avoid plan failure. Cloud NAT and the IN_USE_ADDRESSES quota auto-request remain unused on these clusters until the user does a destroy+recreate to flip the flag. - Fresh deployments after this commit get the full private + NAT design with no extra config required.
Member
|
One additional GKE-specific improvement we could make involves the autoscaling behavior. Instead of the current chart default ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.