Skip to content

add single process oom kill kubeletconfig to GKE nodes #8600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

add single process oom kill kubeletconfig to GKE nodes #8600

wants to merge 1 commit into from
+126 −107

Conversation

upodroid
Copy link
Member

@upodroid upodroid commented Oct 5, 2025

This PR is busy but it does the following

  • use the gke project factory module instead of our custom module
  • The current GKE cluster is route based so we can't use the gke module we use in k8s-infra-prow project.
  • The single_process_oom_kill field hasn't made it to the GKE module yet. Once it does, I'll mark the PR ready for review.

/cc @BenTheElder @ameukam

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 5, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. area/infra Infrastructure management, infrastructure design, code in infra/ area/infra/gcp Issues or PRs related to Kubernetes GCP infrastructure area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 5, 2025
upodroid
upodroid commented Oct 5, 2025
View reviewed changes
infra/gcp/terraform/k8s-infra-prow-build/provider.tf
backend "gcs" {
bucket = "k8s-infra-tf-prow-clusters"
prefix = "k8s-infra-prow-build/prow-build" // $project_name/$cluster_name
bucket = "k8s-infra-terraform"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a state bucket in the k8s-infra-seed project instead of the kubernetes-public

@k8s-infra-ci-robot
Copy link
Contributor

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Plan Error

Show Output 
running 'sh -c' '/atlantis/bin/terraform1.13.3 plan -input=false -refresh -out "/atlantis/repos/kubernetes/k8s.io/8600/default/infra/gcp/terraform/k8s-infra-prow-build/default.tfplan"' in '/atlantis/repos/kubernetes/k8s.io/8600/default/infra/gcp/terraform/k8s-infra-prow-build': exit status 1
module.project.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=2oQ]
module.project.module.project-factory.google_project.main: Refreshing state... [id=projects/k8s-infra-prow-build]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Refreshing state... [id=projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool6-20250327232037500200000001]
module.prow_build_nodepool_c4d_highmem_8_localssd.google_container_node_pool.node_pool: Refreshing state... [id=projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool7-20250528124554315100000001]
module.prow_build_nodepool_c4a_highmem_8_localssd.google_container_node_pool.node_pool: Refreshing state... [id=projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool7-arm64-20250528112313954700000001]
module.project.module.project-factory.module.project_services.google_project_service.project_services["cloudasset.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/cloudasset.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["container.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/container.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/compute.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/bigquery.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/cloudbuild.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["secretmanager.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/secretmanager.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/cloudkms.googleapis.com]
module.project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Refreshing state... [id=k8s-infra-prow-build/artifactregistry.googleapis.com]
google_compute_global_address.grafana_ingress: Refreshing state... [id=projects/k8s-infra-prow-build/global/addresses/grafana-ingress]
google_vmwareengine_network_peering.gvce_peering: Refreshing state... [id=projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project]
module.prow_build_cluster.google_service_account.cluster_node_sa: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com]
google_compute_address.kubernetes_external_secrets_metrics: Refreshing state... [id=projects/k8s-infra-prow-build/regions/us-central1/addresses/kubernetes-external-secrets-metrics]
google_compute_address.boskos_metrics: Refreshing state... [id=projects/k8s-infra-prow-build/regions/us-central1/addresses/boskos-metrics]
google_secret_manager_secret.build_cluster_secrets["prow-build-service-account"]: Refreshing state... [id=projects/k8s-infra-prow-build/secrets/prow-build-service-account]
google_secret_manager_secret.build_cluster_secrets["prow-build-ssh-key-secret-ssh-private"]: Refreshing state... [id=projects/k8s-infra-prow-build/secrets/prow-build-ssh-key-secret-ssh-private]
google_iam_workload_identity_pool.eks_cluster: Refreshing state... [id=projects/k8s-infra-prow-build/locations/global/workloadIdentityPools/prow-eks]
google_secret_manager_secret.build_cluster_secrets["prow-build-ssh-key-secret-ssh-public"]: Refreshing state... [id=projects/k8s-infra-prow-build/secrets/prow-build-ssh-key-secret-ssh-public]
google_monitoring_dashboard.dashboards["build-cluster-capacity-usage.json"]: Refreshing state... [id=projects/773781448124/dashboards/10925237040785467832]
google_monitoring_dashboard.dashboards["prow-build-ephemeral-ssd-experiment.json"]: Refreshing state... [id=projects/773781448124/dashboards/f0163540-a8b7-4618-8308-66652d3d4794]
google_monitoring_dashboard.dashboards["prowjob-resource-usage.json"]: Refreshing state... [id=projects/773781448124/dashboards/10510319052103514664]
google_secret_manager_secret_iam_binding.build_cluster_secret_admins["prow-build-service-account"]: Refreshing state... [id=projects/k8s-infra-prow-build/secrets/prow-build-service-account/roles/secretmanager.admin]
google_secret_manager_secret_iam_binding.build_cluster_secret_admins["prow-build-ssh-key-secret-ssh-private"]: Refreshing state... [id=projects/k8s-infra-prow-build/secrets/prow-build-ssh-key-secret-ssh-private/roles/secretmanager.admin]
google_secret_manager_secret_iam_binding.build_cluster_secret_admins["prow-build-ssh-key-secret-ssh-public"]: Refreshing state... [id=projects/k8s-infra-prow-build/secrets/prow-build-ssh-key-secret-ssh-public/roles/secretmanager.admin]
module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/secretmanager.secretAccessor"]: Refreshing state... [id=k8s-infra-prow-build/roles/secretmanager.secretAccessor]
module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/viewer"]: Refreshing state... [id=k8s-infra-prow-build/roles/viewer]
module.iam.google_project_iam_binding.project_iam_authoritative["default--organizations/758905017065/roles/prow.viewer"]: Refreshing state... [id=k8s-infra-prow-build/organizations/758905017065/roles/prow.viewer]
module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/container.admin"]: Refreshing state... [id=k8s-infra-prow-build/roles/container.admin]
module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/owner"]: Refreshing state... [id=k8s-infra-prow-build/roles/owner]
module.prow_build_cluster.google_project_iam_member.cluster_node_sa_logging: Refreshing state... [id=k8s-infra-prow-build/roles/logging.logWriter/serviceaccount:gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com]
module.prow_build_cluster.google_project_iam_member.cluster_node_sa_monitoring_viewer: Refreshing state... [id=k8s-infra-prow-build/roles/monitoring.viewer/serviceaccount:gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com]
module.prow_build_cluster.google_project_iam_member.cluster_node_sa_monitoring_metricwriter: Refreshing state... [id=k8s-infra-prow-build/roles/monitoring.metricWriter/serviceaccount:gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com]
module.prow_build_cluster.google_bigquery_dataset.prod_usage_metering[0]: Refreshing state... [id=projects/k8s-infra-prow-build/datasets/usage_metering_prow_build]
google_iam_workload_identity_pool_provider.eks_kops: Refreshing state... [id=projects/k8s-infra-prow-build/locations/global/workloadIdentityPools/prow-eks/providers/kops]
google_iam_workload_identity_pool_provider.eks_cluster: Refreshing state... [id=projects/k8s-infra-prow-build/locations/global/workloadIdentityPools/prow-eks/providers/oidc]
module.workload_identity_service_accounts["prow-build"].google_service_account.serviceaccount: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/[email protected]]
module.workload_identity_service_accounts["boskos-janitor"].google_service_account.serviceaccount: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/[email protected]]
module.workload_identity_service_accounts["kubernetes-external-secrets"].google_service_account.serviceaccount: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/kubernetes-external-secrets@k8s-infra-prow-build.iam.gserviceaccount.com]
module.workload_identity_service_accounts["kubernetes-external-secrets"].data.google_iam_policy.workload_identity: Reading...
module.workload_identity_service_accounts["kubernetes-external-secrets"].data.google_iam_policy.workload_identity: Read complete after 0s [id=2784214788]
module.workload_identity_service_accounts["prow-build"].data.google_iam_policy.workload_identity: Reading...
module.workload_identity_service_accounts["prow-build"].data.google_iam_policy.workload_identity: Read complete after 0s [id=1138332750]
module.workload_identity_service_accounts["boskos-janitor"].data.google_iam_policy.workload_identity: Reading...
module.workload_identity_service_accounts["boskos-janitor"].data.google_iam_policy.workload_identity: Read complete after 0s [id=4153730932]
module.prow_build_cluster.google_container_cluster.prod_cluster[0]: Refreshing state... [id=projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build]
module.workload_identity_service_accounts["boskos-janitor"].google_service_account_iam_policy.serviceaccount_iam: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/[email protected]]
module.workload_identity_service_accounts["prow-build"].google_service_account_iam_policy.serviceaccount_iam: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/[email protected]]
module.workload_identity_service_accounts["kubernetes-external-secrets"].google_service_account_iam_policy.serviceaccount_iam: Refreshing state... [id=projects/k8s-infra-prow-build/serviceAccounts/kubernetes-external-secrets@k8s-infra-prow-build.iam.gserviceaccount.com]
module.workload_identity_service_accounts["kubernetes-external-secrets"].google_project_iam_member.project_roles["roles/secretmanager.secretAccessor"]: Refreshing state... [id=k8s-infra-prow-build/roles/secretmanager.secretAccessor/serviceAccount:kubernetes-external-secrets@k8s-infra-prow-build.iam.gserviceaccount.com]
module.sig_node_node_pool_1_n4_highmem_8.google_container_node_pool.nodepool: Refreshing state... [id=projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/sig-node-pool1]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
  - destroy

Terraform planned the following actions, but then encountered a problem:

  # google_vmwareengine_network_peering.gvce_peering will be updated in-place
  ~ resource "google_vmwareengine_network_peering" "gvce_peering" {
      ~ export_custom_routes_with_public_ip = false -> true
        id                                  = "projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project"
      ~ import_custom_routes_with_public_ip = false -> true
        name                                = "peer-with-gcve-project"
        # (13 unchanged attributes hidden)
    }

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be destroyed
  # (because google_container_node_pool.node_pool is not in configuration)
  - resource "google_container_node_pool" "node_pool" {
      - cluster                     = (sensitive value) -> null
      - id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool6-20250327232037500200000001" -> null
      - initial_node_count          = 1 -> null
      - instance_group_urls         = [
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroupManagers/gke-prow-build-pool6-2025032723203750-e834670e-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroupManagers/gke-prow-build-pool6-2025032723203750-da764ec7-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-f/instanceGroupManagers/gke-prow-build-pool6-2025032723203750-2f04babf-grp",
        ] -> null
      - location                    = (sensitive value) -> null
      - managed_instance_group_urls = [
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroups/gke-prow-build-pool6-2025032723203750-e834670e-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroups/gke-prow-build-pool6-2025032723203750-da764ec7-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-f/instanceGroups/gke-prow-build-pool6-2025032723203750-2f04babf-grp",
        ] -> null
      - name                        = "pool6-20250327232037500200000001" -> null
      - name_prefix                 = "pool6-" -> null
      - node_count                  = 2 -> null
      - node_locations              = [
          - "us-central1-b",
          - "us-central1-c",
          - "us-central1-f",
        ] -> null
      - project                     = "k8s-infra-prow-build" -> null
      - version                     = "1.33.4-gke.1036000" -> null

      - autoscaling {
          - location_policy      = "BALANCED" -> null
          - max_node_count       = 5 -> null
          - min_node_count       = 1 -> null
          - total_max_node_count = 0 -> null
          - total_min_node_count = 0 -> null
        }

      - management {
          - auto_repair  = true -> null
          - auto_upgrade = true -> null
        }

      - node_config {
          - disk_size_gb                = 500 -> null
          - disk_type                   = "hyperdisk-balanced" -> null
          - effective_taints            = [
              - {
                  - effect = "PREFER_NO_SCHEDULE"
                  - key    = "spare"
                  - value  = "true"
                },
            ] -> null
          - enable_confidential_storage = false -> null
          - flex_start                  = false -> null
          - image_type                  = "COS_CONTAINERD" -> null
          - labels                      = {} -> null
          - local_ssd_count             = 0 -> null
          - logging_variant             = "DEFAULT" -> null
          - machine_type                = "c4-highmem-8" -> null
          - metadata                    = {
              - "disable-legacy-endpoints" = "true"
            } -> null
          - oauth_scopes                = [
              - "https://www.googleapis.com/auth/cloud-platform",
            ] -> null
          - preemptible                 = false -> null
          - resource_labels             = {
              - "goog-gke-node-pool-provisioning-model" = "on-demand"
            } -> null
          - resource_manager_tags       = {} -> null
          - service_account             = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com" -> null
          - spot                        = false -> null
          - storage_pools               = [] -> null
          - tags                        = [] -> null
            # (5 unchanged attributes hidden)

          - kubelet_config {
              - allowed_unsafe_sysctls                 = [] -> null
              - container_log_max_files                = 0 -> null
              - cpu_cfs_quota                          = false -> null
              - eviction_max_pod_grace_period_seconds  = 0 -> null
              - image_gc_high_threshold_percent        = 0 -> null
              - image_gc_low_threshold_percent         = 0 -> null
              - insecure_kubelet_readonly_port_enabled = "TRUE" -> null
              - max_parallel_image_pulls               = 2 -> null
              - pod_pids_limit                         = 0 -> null
              - single_process_oom_kill                = false -> null
                # (5 unchanged attributes hidden)
            }

          - shielded_instance_config {
              - enable_integrity_monitoring = true -> null
              - enable_secure_boot          = false -> null
            }

          - windows_node_config {
                # (1 unchanged attribute hidden)
            }

          - workload_metadata_config {
              - mode = "GKE_METADATA" -> null
            }
        }

      - upgrade_settings {
          - max_surge       = 1 -> null
          - max_unavailable = 0 -> null
          - strategy        = "SURGE" -> null
        }
    }

  # module.prow_build_nodepool_c4a_highmem_8_localssd.google_container_node_pool.node_pool will be destroyed
  # (because google_container_node_pool.node_pool is not in configuration)
  - resource "google_container_node_pool" "node_pool" {
      - cluster                     = (sensitive value) -> null
      - id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool7-arm64-20250528112313954700000001" -> null
      - initial_node_count          = 1 -> null
      - instance_group_urls         = [
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-a/instanceGroupManagers/gke-prow-build-pool7-arm64-2025052811-0bf73792-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroupManagers/gke-prow-build-pool7-arm64-2025052811-f1ffb9b9-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroupManagers/gke-prow-build-pool7-arm64-2025052811-91499c5d-grp",
        ] -> null
      - location                    = (sensitive value) -> null
      - managed_instance_group_urls = [
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-a/instanceGroups/gke-prow-build-pool7-arm64-2025052811-0bf73792-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroups/gke-prow-build-pool7-arm64-2025052811-f1ffb9b9-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroups/gke-prow-build-pool7-arm64-2025052811-91499c5d-grp",
        ] -> null
      - name                        = "pool7-arm64-20250528112313954700000001" -> null
      - name_prefix                 = "pool7-arm64-" -> null
      - node_count                  = 1 -> null
      - node_locations              = [
          - "us-central1-a",
          - "us-central1-b",
          - "us-central1-c",
        ] -> null
      - project                     = "k8s-infra-prow-build" -> null
      - version                     = "1.32.7-gke.1079000" -> null

      - autoscaling {
          - location_policy      = "BALANCED" -> null
          - max_node_count       = 10 -> null
          - min_node_count       = 1 -> null
          - total_max_node_count = 0 -> null
          - total_min_node_count = 0 -> null
        }

      - management {
          - auto_repair  = true -> null
          - auto_upgrade = true -> null
        }

      - node_config {
          - disk_size_gb                = 100 -> null
          - disk_type                   = "hyperdisk-balanced" -> null
          - effective_taints            = [
              - {
                  - effect = "NO_SCHEDULE"
                  - key    = "kubernetes.io/arch"
                  - value  = "arm64"
                },
            ] -> null
          - enable_confidential_storage = false -> null
          - flex_start                  = false -> null
          - image_type                  = "COS_CONTAINERD" -> null
          - labels                      = {} -> null
          - local_ssd_count             = 0 -> null
          - logging_variant             = "DEFAULT" -> null
          - machine_type                = "c4a-highmem-8-lssd" -> null
          - metadata                    = {
              - "disable-legacy-endpoints" = "true"
            } -> null
          - oauth_scopes                = [
              - "https://www.googleapis.com/auth/cloud-platform",
            ] -> null
          - preemptible                 = false -> null
          - resource_labels             = {
              - "goog-gke-node-pool-provisioning-model" = "on-demand"
            } -> null
          - resource_manager_tags       = {} -> null
          - service_account             = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com" -> null
          - spot                        = false -> null
          - storage_pools               = [] -> null
          - tags                        = [] -> null
            # (5 unchanged attributes hidden)

          - ephemeral_storage_local_ssd_config {
              - data_cache_count = 0 -> null
              - local_ssd_count  = 2 -> null
            }

          - kubelet_config {
              - allowed_unsafe_sysctls                 = [] -> null
              - container_log_max_files                = 0 -> null
              - cpu_cfs_quota                          = false -> null
              - eviction_max_pod_grace_period_seconds  = 0 -> null
              - image_gc_high_threshold_percent        = 0 -> null
              - image_gc_low_threshold_percent         = 0 -> null
              - insecure_kubelet_readonly_port_enabled = "TRUE" -> null
              - max_parallel_image_pulls               = 0 -> null
              - pod_pids_limit                         = 0 -> null
              - single_process_oom_kill                = false -> null
                # (5 unchanged attributes hidden)
            }

          - shielded_instance_config {
              - enable_integrity_monitoring = true -> null
              - enable_secure_boot          = false -> null
            }

          - windows_node_config {
                # (1 unchanged attribute hidden)
            }

          - workload_metadata_config {
              - mode = "GKE_METADATA" -> null
            }
        }

      - upgrade_settings {
          - max_surge       = 1 -> null
          - max_unavailable = 0 -> null
          - strategy        = "SURGE" -> null
        }
    }

  # module.prow_build_nodepool_c4d_highmem_8_localssd.google_container_node_pool.node_pool will be destroyed
  # (because google_container_node_pool.node_pool is not in configuration)
  - resource "google_container_node_pool" "node_pool" {
      - cluster                     = (sensitive value) -> null
      - id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool7-20250528124554315100000001" -> null
      - initial_node_count          = 1 -> null
      - instance_group_urls         = [
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-a/instanceGroupManagers/gke-prow-build-pool7-2025052812455431-48d85bd4-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroupManagers/gke-prow-build-pool7-2025052812455431-f2ab4558-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroupManagers/gke-prow-build-pool7-2025052812455431-2e4cd1ad-grp",
        ] -> null
      - location                    = (sensitive value) -> null
      - managed_instance_group_urls = [
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-a/instanceGroups/gke-prow-build-pool7-2025052812455431-48d85bd4-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroups/gke-prow-build-pool7-2025052812455431-f2ab4558-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroups/gke-prow-build-pool7-2025052812455431-2e4cd1ad-grp",
        ] -> null
      - name                        = "pool7-20250528124554315100000001" -> null
      - name_prefix                 = "pool7-" -> null
      - node_count                  = 15 -> null
      - node_locations              = [
          - "us-central1-a",
          - "us-central1-b",
          - "us-central1-c",
        ] -> null
      - project                     = "k8s-infra-prow-build" -> null
      - version                     = "1.32.7-gke.1016000" -> null

      - autoscaling {
          - location_policy      = "BALANCED" -> null
          - max_node_count       = 80 -> null
          - min_node_count       = 1 -> null
          - total_max_node_count = 0 -> null
          - total_min_node_count = 0 -> null
        }

      - management {
          - auto_repair  = true -> null
          - auto_upgrade = true -> null
        }

      - node_config {
          - disk_size_gb                = 100 -> null
          - disk_type                   = "hyperdisk-balanced" -> null
          - effective_taints            = [] -> null
          - enable_confidential_storage = false -> null
          - flex_start                  = false -> null
          - image_type                  = "COS_CONTAINERD" -> null
          - labels                      = {} -> null
          - local_ssd_count             = 0 -> null
          - logging_variant             = "DEFAULT" -> null
          - machine_type                = "c4d-highmem-8-lssd" -> null
          - metadata                    = {
              - "disable-legacy-endpoints" = "true"
            } -> null
          - oauth_scopes                = [
              - "https://www.googleapis.com/auth/cloud-platform",
            ] -> null
          - preemptible                 = false -> null
          - resource_labels             = {
              - "goog-gke-node-pool-provisioning-model" = "on-demand"
            } -> null
          - resource_manager_tags       = {} -> null
          - service_account             = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com" -> null
          - spot                        = false -> null
          - storage_pools               = [] -> null
          - tags                        = [] -> null
            # (5 unchanged attributes hidden)

          - ephemeral_storage_local_ssd_config {
              - data_cache_count = 0 -> null
              - local_ssd_count  = 1 -> null
            }

          - kubelet_config {
              - allowed_unsafe_sysctls                 = [] -> null
              - container_log_max_files                = 0 -> null
              - cpu_cfs_quota                          = false -> null
              - eviction_max_pod_grace_period_seconds  = 0 -> null
              - image_gc_high_threshold_percent        = 0 -> null
              - image_gc_low_threshold_percent         = 0 -> null
              - insecure_kubelet_readonly_port_enabled = "TRUE" -> null
              - max_parallel_image_pulls               = 0 -> null
              - pod_pids_limit                         = 0 -> null
              - single_process_oom_kill                = false -> null
                # (5 unchanged attributes hidden)
            }

          - shielded_instance_config {
              - enable_integrity_monitoring = true -> null
              - enable_secure_boot          = false -> null
            }

          - windows_node_config {
                # (1 unchanged attribute hidden)
            }

          - workload_metadata_config {
              - mode = "GKE_METADATA" -> null
            }
        }

      - upgrade_settings {
          - max_surge       = 1 -> null
          - max_unavailable = 0 -> null
          - strategy        = "SURGE" -> null
        }
    }

Plan: 0 to add, 1 to change, 3 to destroy.
╷
│ Error: Invalid function argument
│ 
│   on .terraform/modules/prod_amd_pool/modules/gke-node-pool/main.tf line 208, in resource "google_container_node_pool" "main":
│  208:           insecure_kubelet_readonly_port_enabled = upper(tostring(kubelet_config.value.insecure_kubelet_readonly_port_enabled))
│     ├────────────────
│     │ kubelet_config.value.insecure_kubelet_readonly_port_enabled is null
│ 
│ Invalid value for "str" parameter: argument must not be null.
╵
╷
│ Error: Invalid function argument
│ 
│   on .terraform/modules/prod_arm_pool/modules/gke-node-pool/main.tf line 208, in resource "google_container_node_pool" "main":
│  208:           insecure_kubelet_readonly_port_enabled = upper(tostring(kubelet_config.value.insecure_kubelet_readonly_port_enabled))
│     ├────────────────
│     │ kubelet_config.value.insecure_kubelet_readonly_port_enabled is null
│ 
│ Invalid value for "str" parameter: argument must not be null.
╵
╷
│ Error: Invalid function argument
│ 
│   on .terraform/modules/prod_intel_pool/modules/gke-node-pool/main.tf line 208, in resource "google_container_node_pool" "main":
│  208:           insecure_kubelet_readonly_port_enabled = upper(tostring(kubelet_config.value.insecure_kubelet_readonly_port_enabled))
│     ├────────────────
│     │ kubelet_config.value.insecure_kubelet_readonly_port_enabled is null
│ 
│ Invalid value for "str" parameter: argument must not be null.
╵

@BenTheElder
Copy link
Member

Not super familiar with the terraform project but we might have to file an issue about the kubelet option

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ameukam ameukam Awaiting requested review from ameukam

@BenTheElder BenTheElder Awaiting requested review from BenTheElder

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/infra/gcp Issues or PRs related to Kubernetes GCP infrastructure area/infra Infrastructure management, infrastructure design, code in infra/ area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@upodroid @k8s-ci-robot @k8s-infra-ci-robot @BenTheElder