diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/README.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/README.md index 29b986eb085..a5e04f36490 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/README.md +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/README.md @@ -2,7 +2,7 @@ The code in `k8s-infra-gcp-gcve` sets up the infra required to allow prow jobs to create VMs on vSphere, e.g. to allow testing of the [Cluster API provider vSphere (CAPV)](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere). -![Overview](./docs/images/overview.jpg) +![Overview](./docs/images/GVCE.drawio.png) Prow container settings are managed outside of this folder, but understanding high level components could help to understand how the `k8s-infra-gcp-gcve` is set up and consumed. @@ -14,26 +14,101 @@ More specifically, to allow prow jobs to create VM on vSphere, a few resources a - a vSphere folder and a vSphere resources pool where to run VMs during a test. - a reserved IP range to be used for the test e.g. for the kube vip load balancer in a CAPV cluster (VM instead will get IPs via DHCP). -Also, the network of the prow container is going to be paired to the VMware engine network, thus +Also, the network of the prow container is going to be peered to the VMware engine network, thus allowing access to both the GCVE management network and the NSX-T network where all the VM are running. -The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. There are ther main components of this infrastracture: +The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. +These are the main components of this infrastructure: -The terraform manifest in this folder, which is applied by test-infra automation (Atlantis), uses the GCP terraform provider for creating. +The terraform manifest in this folder uses the GCP terraform provider for creating. - A VMware Engine instance - The network infrastructure required for vSphere and for allowing communication between vSphere and Prow container. - - The network used is `192.168.0.32/21` + - The network used is `192.168.32.0/21` - Usable Host IP Range: `192.168.32.1 - 192.168.39.254` - DHCP Range: `192.168.32.11 - 192.168.33.255` - IPPool for 40 Projects having 16 IPs each: `192.168.35.0 - 192.168.37.127` - The network infrastructure used for maintenance. +See [terraform](./docs/terraform.md) for prerequisites. + +When ready: + +```sh +terraform init +terraform plan # Check diff +terraform apply +``` + See inline comments for more details. -The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance pourposes. See +The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance purposes. See - [maintenance-jumphost](./maintenance-jumphost/README.md) -The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templetes, folders, +The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templates, folders, resource pools and other vSphere components required when running tests. See: -- [vsphere](./vsphere/README.md) +- [vsphere](./vsphere/README.md) + +# Working around network issues + +There are two issues that can exist: + +## Existing limitation: maximum number of 64 connections/requests to an internet endpoint + +Workaround: + +* Route traffic via 192.168.32.8 (see [NSX Gateway VM](./nsx-gateway/)) + * which tunnels via the maintenance jumphost + +## Packages are dropped without any hop + +Example: `mtr -T -P 443 1.2.3.4` shows no hop at all (not even the gateway) + +It could be that NSX-T started dropping traffic to that endpoint. + +The workaround is documented in [Disabling NSX-T Firewalls](./vsphere/README.md#disabling-nsx-t-firewalls) + +## Setting the right MTU + +Setting the right MTU is important to not run into connectivity issues due to dropped packages. +For the workload network `k8s-ci` in NSX-T, the correct MTU is configured in [nsx-t.tf](./vsphere/nsx-t.tf) and was determined by using `ping` e.g. via [this bash script](https://gist.githubusercontent.com/penguin2716/e3c2186d0da6b96845fd54a275a2cd71/raw/e4b45c33c99c6c03b200186bf2cb6b1af3d806f5/find_max_mtu.sh). + +# Uploading OVA's + +Pre-created OVA's are available at [Cluster API Provider vSphere releases](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases?q=%22VM+templates%22&expanded=true). + +There is a script which automates the download from Github, concatenate files (if necessary) and upload to vSphere. + +First we have to identify the link or links for an OVA. + +E.g. from [templates/v1.33.0](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/tag/templates%2Fv1.33.0): + +* https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/download/templates%2Fv1.33.0/ubuntu-2204-kube-v1.33.0.ova-part-aa +* https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/download/templates%2Fv1.33.0/ubuntu-2204-kube-v1.33.0.ova-part-ab + +Then we have to set credentials and run the script with both urls as parameters: + +```sh +export GOVC_URL="$(gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='get(vcenter.fqdn)')" +export GOVC_USERNAME="solution-user-01@gve.local" +export GOVC_PASSWORD="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve --username=solution-user-01@gve.local --location=us-central1-a --format='get(password)')" + +infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/upload-ova.sh https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/download/templates%2Fv1.33.0/ubuntu-2204-kube-v1.33.0.ova-part-aa https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/download/templates%2Fv1.33.0/ubuntu-2204-kube-v1.33.0.ova-part-ab +``` + +# Recreating the whole environment + +Deleting a Google Cloud VMware Engine (GCVE) Private Cloud results in a 1 week freeze of the old environment. +Because of that it is not possible to immediately recreate the environment using the same configuration. + +If recreation needs to be done, we have to change the following: + +* The management cidr (`192.168.31.0/24`, search for all occurencies of `192.168.31`) +* The name of the private cloud (search for all occurencies of `k8s-gcp-gcve`) + +For deleting the old environment: +* Use the UI for [Private Clouds](https://console.cloud.google.com/vmwareengine/privateclouds?project=broadcom-451918) to start the deletion process + * Note: With starting the deletion process, the Private Cloud will also not be billed anymore. +* Afterwards setup terraform as described above and remove the private cloud from the state: + * `terraform state rm google_vmwareengine_private_cloud.vsphere-cluster` +* Finally use the above documentation to re-provision VMware Engine diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/boskos.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/boskos.md index 01af5fddde8..5c4337e3da0 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/boskos.md +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/boskos.md @@ -1,6 +1,6 @@ # Boskos -Boskos support resources of type `gcve-vsphere-project` to allow each test run to use a subset of vSphere resources. +Boskos resources of type `gcve-vsphere-project` allow each test run to use a subset of vSphere resources. Boskos configuration is split in three parts: diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/GVCE.drawio b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/GVCE.drawio deleted file mode 100644 index 90e82e1a89f..00000000000 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/GVCE.drawio +++ /dev/null @@ -1,216 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/GVCE.drawio.png b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/GVCE.drawio.png new file mode 100644 index 00000000000..45c59c01e42 Binary files /dev/null and b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/GVCE.drawio.png differ diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/overview.jpg b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/overview.jpg deleted file mode 100644 index 5eb9eeff7e3..00000000000 Binary files a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/images/overview.jpg and /dev/null differ diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/terraform.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/terraform.md index 7fc39d902cf..95a08ae50d3 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/terraform.md +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/terraform.md @@ -2,13 +2,13 @@ See [README.md](https://github.com/kubernetes/k8s.io/tree/main/infra/gcp/terraform) for a general intro about using terraform in k8s.io. -In order to apply terraform manifests you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need. +In order to apply terraform manifests you must be enabled to use the "broadcom-451918" project, please reach out to [owners](../OWNERS) in case of need. Quick reference: Go to the folder of interest -- [maintenance-jumphost](../maintenance-jumphost/) -- [vsphere](../vsphere/) +- [maintenance-jumphost](../maintenance-jumphost/README.md) +- [vsphere](../vsphere/README.md) Note: the terraform script in the top folder is usually managed by test-infra automation (Atlantis); we don't have to run it manually. @@ -19,26 +19,35 @@ You can use terraform from your local workstation or via a docker container prov docker run -it --rm -v $(pwd):/workspace --entrypoint=/bin/bash gcr.io/k8s-staging-infra-tools/k8s-infra:v20241217-f8b07a049 ``` -From your local workstatin / from inside the terraform container: +From your local workstation / from inside the terraform container: Login to GCP to get an authentication token to use with terraform. ```bash gcloud auth application-default login +gcloud auth login +gcloud config set project broadcom-451918 ``` Ensure all the env variables expected by the terraform manifest you are planning to run are set: -- [vsphere](../vsphere/) +- [vsphere](../vsphere/README.md) + +Ensure the right terraform version expected by the terraform manifest you are planning to run is installed (Note: this requires `tfswitch` which is pre-installed in the docker image. In case of version mismatches, terraform will make you know): + +```bash +cd infra/gcp/terraform/k8s-infra-gcp-gcve/ +tfswitch +``` Additionally, if applying the vsphere terraform manifest, use the following script to generate `/etc/hosts` entries for vSphere and NSX. ```sh -gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn' +gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn' ``` -Add those entry to `/etc/hosts`. +Add those entries to `/etc/hosts`. -At this point you are ready to start using `terraform init`, `terraform plan`, `terraform apply` etc. +At this point you are ready to start using `terraform init`, `terraform plan`, `terraform apply` etc. Notes: - Terraform state is stored in a gcs bucket with name `k8s-infra-tf-gcp-gcve`, with a folder for each one of the terraform scripts managed in the `k8s-infra-gcp-gcve` folder (gcve, gcve-vcenter, maintenance-jumphost). \ No newline at end of file diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wiregard.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wireguard.md similarity index 74% rename from infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wiregard.md rename to infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wireguard.md index 3bce6adaf08..9e90e0d3d57 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wiregard.md +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wireguard.md @@ -1,14 +1,16 @@ -# Wiregard +# Wireguard -Wiregard is used to get a secure and convenient access through the maintenace jump host VM. +Wireguard is used to get a secure and convenient access through the maintenace jump host VM. -In order to use wiregard you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need. +In order to use wireguard you must be enabled to use the "broadcom-451918" project, please reach out to [owners](../OWNERS) in case of need. -It is also required to first setup things both on on your local machine and on the GCP side -following instruction below. +It is also required to first setup things both on your local machine and on the GCP side +following the instruction below. Install wireguard following one of the methods described in https://www.wireguard.com/install/. +Note: On OSX to use the commandline tool, installation via `brew` is necessary. + Generate a wireguard keypair using `wg`. ```sh @@ -58,7 +60,7 @@ EOF Then create new version of the `maintenance-vm-wireguard-config` by appending this entry at the end of the current value [here](https://console.cloud.google.com/security/secret-manager/secret/maintenance-vm-wireguard-config/versions?project=broadcom-451918). -Additionally, if the jumphost VM is up, you might want to add it to the wiregard configuration in the current VM (it is also possible to recreate the jumphost VM, but this is going to change the wireguard enpoint also for other users). +Additionally, if the jumphost VM is up, you might want to add it to the wireguard configuration in the current VM (it is also possible to recreate the jumphost VM, but this is going to change the wireguard enpoint also for other users). ```sh gcloud compute ssh maintenance-jumphost --zone us-central1-f @@ -79,14 +81,15 @@ MTU = 1360 [Peer] PublicKey = $(gcloud secrets versions access --secret maintenance-vm-wireguard-pubkey latest) -AllowedIPs = 192.168.30.0/24, 192.168.32.0/21 +AllowedIPs = 192.168.31.0/24, 192.168.32.0/21 Endpoint = $(gcloud compute instances list --format='get(networkInterfaces[0].accessConfigs[0].natIP)' --filter='name=maintenance-jumphost'):51820 PersistentKeepalive = 25 EOF ``` You can then either -- import this file to the wireguard UI (after this, you can remove the file from disk). + +- import this file to the wireguard UI (after this, you can remove the file from disk) and activate or deactivate the connection. - use the file with the wireguard CLI e.g. `wg-quick up ~/wg0.conf`, and when finished `wg-quick down ~/wg0.conf` ## Additional settings @@ -94,7 +97,7 @@ You can then either Generate `/etc/hosts` entries for vSphere and NSX; this is required to run the vSphere terraform scripts and it will also make the vSphere and NSX UI to work smootly. ```sh -gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn' +gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn' ``` -Add those entry to `/etc/hosts`. \ No newline at end of file +Add those entries to `/etc/hosts`. \ No newline at end of file diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/iam.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/iam.tf index d9b4d44a9f8..e5c1ca2c7da 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/iam.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/iam.tf @@ -14,6 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Ensures admin access for groups and secret access for prow. module "iam" { source = "terraform-google-modules/iam/google//modules/projects_iam" version = "~> 8.1" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/main.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/main.tf index 8dc1c08389a..66b17c38cc5 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/main.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/main.tf @@ -14,19 +14,17 @@ See the License for the specific language governing permissions and limitations under the License. */ -locals { - project_id = "broadcom-451918" -} - data "google_project" "project" { - project_id = local.project_id + project_id = var.project_id } +# Enables all required APIs for this project. resource "google_project_service" "project" { project = data.google_project.project.id for_each = toset([ "compute.googleapis.com", + "essentialcontacts.googleapis.com", "secretmanager.googleapis.com", "vmwareengine.googleapis.com" ]) diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/README.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/README.md index 1aa0eda08e6..9001b7a3435 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/README.md +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/README.md @@ -3,11 +3,14 @@ The maintenance jump host is a VM hosting a wireguard instance for secure and convenient access to vSphere and NSX from local machines. -Before using wiregard it is required to first setup things both on on your local machine and on the GCP side. -see [wireguard](../docs/wiregard.md) +Also it is used to tunnel traffic from workload VMs to the internet. +See [NSX-Gateway](../nsx-gateway/README.md) for more information. -The maintenance jump host VM is not required to be always up & running and it can also be recreated if necessary; however, by doing so the IP address of the VM will change and all the -local machine config have to be updated accordingly. +Before using wireguard it is required to first setup things both on your local machine and on the GCP side. +see [wireguard](../docs/wireguard.md) + +The maintenance jump host VM can be recreated if necessary; however, by doing so the IP address of the VM will change and all the +local machine config and the [NSX-Gateway](../nsx-gateway/README.md) have to be updated accordingly. To check if maintenance jump host VM is already up and running, look for the `maintenance-jumphost` VM into the google cloud console, compute engine, VM instances ([link](https://console.cloud.google.com/compute/instances?project=broadcom-451918)). diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/jumphost.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/jumphost.tf index 5c89420e1c3..c767c1761a9 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/jumphost.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/jumphost.tf @@ -14,12 +14,21 @@ See the License for the specific language governing permissions and limitations under the License. */ -locals { - project_id = "broadcom-451918" +variable "project_id" { + description = "The project ID to use for the gcve cluster." + default = "broadcom-451918" + type = string } +# Read the secret from Secret Manager which contains the wireguard server configuration. +data "google_secret_manager_secret_version_access" "wireguard-config" { + project = var.project_id + secret = "maintenance-vm-wireguard-config" +} + +# Create the maintenance jumphost which runs SSH and a wireguard server. resource "google_compute_instance" "jumphost" { - project = local.project_id + project = var.project_id name = "maintenance-jumphost" machine_type = "f1-micro" zone = "us-central1-f" @@ -33,7 +42,7 @@ resource "google_compute_instance" "jumphost" { network_interface { network = "maintenance-vpc-network" subnetwork = "maintenance-subnet" - subnetwork_project = local.project_id + subnetwork_project = var.project_id access_config { network_tier = "STANDARD" } @@ -43,8 +52,3 @@ resource "google_compute_instance" "jumphost" { user-data = templatefile("${path.module}/cloud-config.yaml.tftpl", { wg0 = base64encode(data.google_secret_manager_secret_version_access.wireguard-config.secret_data) }) } } - -data "google_secret_manager_secret_version_access" "wireguard-config" { - project = local.project_id - secret = "maintenance-vm-wireguard-config" -} diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/provider.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/provider.tf index a522b417ddf..9c64bf35783 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/provider.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/provider.tf @@ -21,6 +21,7 @@ This file defines: */ terraform { + required_version = "1.10.5" backend "gcs" { bucket = "k8s-infra-tf-gcp-gcve" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/README.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/README.md new file mode 100644 index 00000000000..7ab390e16e9 --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/README.md @@ -0,0 +1,45 @@ +# NSX Gateway + +The NSX Gateway is a VM which funcions as network gateway for all workload VMs. + +It is used to route all internet facing traffic from workload VMs via the [Maintenance jumphost](../maintenance-jumphost/README.md). + +This is to workaround an issue of internet traffic, which results in TCP connection errors. + +To provision the NSX Gateway VM. + +See [terraform](../docs/terraform.md) for prerequisites. + +See [wireguard](../docs/wireguard.md for prerequisites. + +```sh +terraform init +terraform apply +``` + +The wireguard config will look like + +```ini +[Interface] +PrivateKey = ... +Address = 192.168.29.6/24 +PostUp = iptables -t nat -I POSTROUTING -o wg0 -j MASQUERADE +PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE + +[Peer] +Endpoint = 192.168.28.3:51820 +PublicKey = ... +PersistentKeepalive = 25 +# all except private networks +AllowedIPs = 0.0.0.0/5, 8.0.0.0/7, 11.0.0.0/8, 12.0.0.0/6, 16.0.0.0/4, 32.0.0.0/3, 64.0.0.0/2, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/12, 172.32.0.0/11, 172.64.0.0/10, 172.128.0.0/9, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/9, 192.128.0.0/11, 192.160.0.0/13, 192.169.0.0/16, 192.170.0.0/15, 192.172.0.0/14, 192.176.0.0/12, 192.192.0.0/10, 193.0.0.0/8, 194.0.0.0/7, 196.0.0.0/6, 200.0.0.0/5, 208.0.0.0/4, 224.0.0.0/3 +``` + +To get SSH access to the VM, redeploy using: + +```sh + export TF_VAR_ssh_public_key="ssh-rsa ..." +terraform taint vsphere_virtual_machine.gateway_vm +terraform apply +``` + +Note: Redeployment causes connection issues for running CI jobs. diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/cloud-config.yaml.tftpl b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/cloud-config.yaml.tftpl new file mode 100644 index 00000000000..d1c85cc9fd8 --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/cloud-config.yaml.tftpl @@ -0,0 +1,29 @@ +#cloud-config + +write_files: +- path: /etc/wireguard/wg0.conf + content: "${wg0}" + encoding: b64 + permissions: "0600" + +- path: /etc/sysctl.d/10-wireguard.conf + content: | + net.ipv4.ip_forward = 1 + +users: +- name: ubuntu + primary_group: ubuntu + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + groups: sudo, wheel + ssh_import_id: None + lock_passwd: true + ssh_authorized_keys: + - "${ssh_public_key}" + +runcmd: +- apt-get update +- apt install wireguard -q -y +- sysctl -p /etc/sysctl.d/10-wireguard.conf +- systemctl enable wg-quick@wg0 +- systemctl start wg-quick@wg0 diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/gateway.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/gateway.tf new file mode 100644 index 00000000000..7fb97215172 --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/gateway.tf @@ -0,0 +1,76 @@ +/* +Copyright 2025 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +# Read the secret from Secret Manager which contains the wireguard server configuration. +data "google_secret_manager_secret_version_access" "wireguard-config" { + project = var.project_id + secret = "nsx-gateway-vm-wireguard-config" +} + +resource "vsphere_virtual_machine" "gateway_vm" { + name = "gateway-vm" + resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id + datastore_id = data.vsphere_datastore.datastore.id + num_cpus = 2 + memory = 2048 + guest_id = "ubuntu64Guest" + network_interface { + network_id = data.vsphere_network.network.id + } + disk { + label = "Hard Disk 1" + size = 20 + } + clone { + template_uuid = data.vsphere_virtual_machine.template.id + } + extra_config = { + "guestinfo.metadata" = base64encode(file("${path.module}/metadata.yaml")) + "guestinfo.userdata" = base64encode(templatefile( + "${path.module}/cloud-config.yaml.tftpl", + { + wg0 = base64encode(data.google_secret_manager_secret_version_access.wireguard-config.secret_data) + ssh_public_key = var.ssh_public_key + } + )) + "guestinfo.metadata.encoding" ="base64" + "guestinfo.userdata.encoding" ="base64" + } +} + +data "vsphere_virtual_machine" "template" { + name = "/Datacenter/vm/prow/templates/ubuntu-2404-kube-v1.33.0" + datacenter_id = data.vsphere_datacenter.datacenter.id +} + +data "vsphere_datacenter" "datacenter" { + name = var.vsphere_datacenter +} + +data "vsphere_datastore" "datastore" { + name = var.vsphere_datastorename + datacenter_id = data.vsphere_datacenter.datacenter.id +} + +data "vsphere_compute_cluster" "cluster" { + name = var.vsphere_cluster + datacenter_id = data.vsphere_datacenter.datacenter.id +} + +data "vsphere_network" "network" { + name = var.vsphere_network_name + datacenter_id = data.vsphere_datacenter.datacenter.id +} \ No newline at end of file diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/metadata.yaml b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/metadata.yaml new file mode 100644 index 00000000000..6a2a3e8ab0f --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/metadata.yaml @@ -0,0 +1,19 @@ +instance-id: gateway-vm +local-hostname: gateway-vm +network: + version: 2 + ethernets: + nics: + match: + name: "e*" + mtu: "1420" + dhcp4: false + addresses: + - 192.168.32.8/21 + nameservers: + addresses: [192.168.31.234] + routes: + - to: default + via: 192.168.32.1 + - to: 192.168.31.234 + via: 192.168.32.1 diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/provider.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/provider.tf new file mode 100644 index 00000000000..4c7d49fb03b --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/provider.tf @@ -0,0 +1,52 @@ +/* +Copyright 2025 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +/* +This file defines: +- Required provider versions +- Storage backend details +*/ + +terraform { + required_version = "1.10.5" + + backend "gcs" { + bucket = "k8s-infra-tf-gcp-gcve" + prefix = "k8s-infra-gcp-gcve-vcenter-gateway" + } + + required_providers { + google = { + source = "hashicorp/google" + version = "~> 6.34.1" + } + google-beta = { + source = "hashicorp/google-beta" + version = "~> 6.34.1" + } + vsphere = { + source = "vmware/vsphere" + version = "~> 2.13.0" + } + } +} + +# Setup credentials to vSphere. +provider "vsphere" { + user = var.vsphere_user + password = var.vsphere_password + vsphere_server = var.vsphere_server +} diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/variables.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/variables.tf new file mode 100644 index 00000000000..166c08778e5 --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/nsx-gateway/variables.tf @@ -0,0 +1,73 @@ +/* +Copyright 2025 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +variable "project_id" { + description = "The project ID to use for the gcve cluster." + default = "broadcom-451918" + type = string +} + +# solution admin user from GCVE +# xref: https://cloud.google.com/vmware-engine/docs/private-clouds/howto-elevate-privilege +variable "vsphere_user" { + type = string +} + +variable "vsphere_password" { + type = string +} + +variable "vsphere_server" { + type = string +} + +# This DNS Server was created by GCVE and can be found in GCVE's summary page. +# xref: https://console.cloud.google.com/vmwareengine/privateclouds/us-central1-a/k8s-gcp-gcve/management-appliances?project=broadcom-451918 +variable "gcve_dns_server" { + type = string + default = "192.168.31.234" +} + +# This is the name of the Datacenter created by GCVE +variable "vsphere_datacenter" { + type = string + default = "Datacenter" +} + +# This is the name of the VMware Engine Private Cloud created via ../vmware-engine.tf +variable "vsphere_cluster" { + type = string + default = "k8s-gcve-cluster" +} + +# This is the name of the Datastore created by GCVE +variable "vsphere_datastorename" { + type = string + default = "vsanDatastore" +} + +# This is the name of the Network which will be created in NSX-T +variable "vsphere_network_name" { + type = string + default = "k8s-ci" +} + +# This is the public key which allows ssh access to the vm +variable "ssh_public_key" { + type = string + default = "" +} + diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/provider.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/provider.tf index 88189b24162..32d504d683d 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/provider.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/provider.tf @@ -22,12 +22,12 @@ This file defines: terraform { required_version = "1.10.5" + backend "gcs" { bucket = "k8s-infra-tf-gcp-gcve" prefix = "k8s-infra-gcp-gcve" } - required_providers { google = { source = "hashicorp/google" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/variables.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/variables.tf index 2d249f9a478..c2a473e9034 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/variables.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/variables.tf @@ -17,4 +17,5 @@ limitations under the License. variable "project_id" { description = "The project ID to use for the gcve cluster." default = "broadcom-451918" + type = string } diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vmware-engine.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vmware-engine.tf index 4afce598bd9..554b0ac298d 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vmware-engine.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vmware-engine.tf @@ -14,15 +14,15 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Creates the VMware Engine Private Cloud which is a vSphere Cluster including NSX-T. resource "google_vmwareengine_private_cloud" "vsphere-cluster" { location = "us-central1-a" - name = "k8s-gcp-gcve-pc" + name = "k8s-gcp-gcve" project = var.project_id description = "k8s Community vSphere Cluster for CI." - # TODO(chrischdi): figure out discount and switch to STANDARD - type = "TIME_LIMITED" + type = "STANDARD" network_config { - management_cidr = "192.168.30.0/24" + management_cidr = "192.168.31.0/24" vmware_engine_network = google_vmwareengine_network.vsphere-network.id } @@ -30,12 +30,12 @@ resource "google_vmwareengine_private_cloud" "vsphere-cluster" { cluster_id = "k8s-gcve-cluster" node_type_configs { node_type_id = "standard-72" - # TODO: node_count 1 is for the TIME_LIMITED version. Change to `3`. - node_count = 1 + node_count = 3 } } } +# Creates the VMware Engine Network for the Private Cloud. resource "google_vmwareengine_network" "vsphere-network" { name = "k8s-gcp-gcve-network" project = var.project_id @@ -43,17 +43,19 @@ resource "google_vmwareengine_network" "vsphere-network" { location = "global" } +# Creates the Network Policy to allow created virtual machines to reach out to the internet. resource "google_vmwareengine_network_policy" "external-access-rule-np" { name = "k8s-gcp-gcve-network-policy" project = var.project_id location = "us-central1" - edge_services_cidr = "192.168.31.0/26" + edge_services_cidr = "192.168.27.0/26" vmware_engine_network = google_vmwareengine_network.vsphere-network.id internet_access { enabled = true } } +# Creates the Peering to the prow cluster to allow Pods running in Prow to access vCenter and created VMs in vSphere. resource "google_vmwareengine_network_peering" "prow_peering" { name = "peer-with-k8s-infra-prow-build" project = var.project_id @@ -64,12 +66,14 @@ resource "google_vmwareengine_network_peering" "prow_peering" { import_custom_routes_with_public_ip = false } +# Creates a maintenance network used for creating Google Compute VM(s) for setup or debugging purposes via ssh or wireguard VPN. resource "google_compute_network" "maintenance-vpc" { name = "maintenance-vpc-network" project = var.project_id auto_create_subnetworks = false } +# Creates the Subnet for the above maintenance network. resource "google_compute_subnetwork" "maintenance-subnet" { name = "maintenance-subnet" project = var.project_id @@ -78,6 +82,7 @@ resource "google_compute_subnetwork" "maintenance-subnet" { network = google_compute_network.maintenance-vpc.id } +# Creates the Peering to the maintenance network to maintenance VMs to access vCenter and created VMs in vSphere. resource "google_vmwareengine_network_peering" "maintenance_peering" { name = "peer-with-maintenance-vpc-network" description = "Peering with maintenance vpc network" @@ -87,6 +92,8 @@ resource "google_vmwareengine_network_peering" "maintenance_peering" { vmware_engine_network = google_vmwareengine_network.vsphere-network.id } +# Creates the firewall rules for VMs running in the maintenance network so they can be accessed +# via SSH or to expose wireguard VPN. resource "google_compute_firewall" "maintenance-firewall-internet" { name = "maintenance-firewall-internet" project = var.project_id @@ -105,6 +112,8 @@ resource "google_compute_firewall" "maintenance-firewall-internet" { } } +# Creates the firewall rule to allow any traffic from the maintenance subnet to +# the VMware Engine network or the internet. resource "google_compute_firewall" "maintenance-firewall-internal" { name = "maintenance-firewall-internal" project = var.project_id diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/README.md b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/README.md index 082c9b80c44..30c098ae7a2 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/README.md +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/README.md @@ -2,27 +2,36 @@ The VMware Engine instance provides a vSphere cluster with NSX-T. -The terraform manifest in this folder can be used to setup e.g. content libraries, templetes, folders, resource pools and other vSphere settings required when running tests. +The terraform manifest in this folder can be used to setup e.g. content libraries, templates, folders, resource pools and other vSphere settings required when running tests. See [terraform](../docs/terraform.md) for prerequisites. -The terraform manifests in this folder requires following env variables: +The first time after creating the VMware Engine Private Cloud we have to reset the solution-user credentials: + +```sh +gcloud vmware private-clouds vcenter credentials reset --private-cloud=k8s-gcp-gcve --username=solution-user-01@gve.local --location=us-central1-a +``` + +The terraform manifests in this folder require following env variables to be set: ```sh export TF_VAR_vsphere_user=solution-user-01@gve.local - export TF_VAR_vsphere_password="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve-pc --username=solution-user-01@gve.local --location=us-central1-a --format='get(password)')" - export TF_VAR_vsphere_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(vcenter.fqdn)')" + export TF_VAR_vsphere_password="$(gcloud vmware private-clouds vcenter credentials describe --private-cloud=k8s-gcp-gcve --username=solution-user-01@gve.local --location=us-central1-a --format='get(password)')" + export TF_VAR_vsphere_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='get(vcenter.fqdn)')" export TF_VAR_nsxt_user=admin - export TF_VAR_nsxt_password="$(gcloud vmware private-clouds nsx credentials describe --private-cloud k8s-gcp-gcve-pc --location us-central1-a --format='get(password)')" - export TF_VAR_nsxt_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='get(nsx.fqdn)')" + export TF_VAR_nsxt_password="$(gcloud vmware private-clouds nsx credentials describe --private-cloud k8s-gcp-gcve --location us-central1-a --format='get(password)')" + export TF_VAR_nsxt_server="$(gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='get(nsx.fqdn)')" ``` -Note: solution-user-01@gve.local user is creating automatically in a VMware Engine instance; +Note: solution-user-01@gve.local user gets created automatically in a VMware Engine instance; we are using it to set up vSphere and create a dedicate user for prow CI (with limited permissions). +For more information see [VMware Engine documentation](https://cloud.google.com/vmware-engine/docs/private-clouds/howto-elevate-privilege). -Also, the terraform manifests in this folder requires `/etc/hosts` entries for vSphere and NSX +Also, the terraform manifests in this folder require `/etc/hosts` entries for vSphere and NSX (see the [terraform](../docs/terraform.md)). +As well as connectivity to vSphere and NSX-T via e.g. [wireguard](../docs/wireguard.md). + Due to missing features in the terraform provider, user and other IAM configuration must be managed with dedicated scripts, the following script needs to be run before terraform apply: Run a fist script to create the prow-ci-user@gve.local user to be used for prow CI. @@ -71,18 +80,29 @@ At the end following secrets should exist: As a final step it is required to setup Boskos resources of type `gcve-vsphere-project` to allow each test run to use a subset of vSphere resources. See [Boskos](../docs/boskos.md). +# Disabling NSX-T Firewalls + +Lastly we need to login to NSX and disable the distributed and gateway firewalls, so they don't accidentially block any traffic. Disabling them is fine because we don't expose endpoints to the internet. + +Login to http://nsx-434005.f63e615b.us-central1.gve.goog (see instructions below). + +Navigate to Security (top) -> Distributed Firewall (left) -> Settings (tab), turn `Distributed Firewall Service` to `Off`, acknowledge dialog which pops up. + +Navigate to Security (top) -> Gateway Firewall (left) -> Settings (tab), turn off all entries at column `Gateway Firewall +(Primary)`. + # Accessing vSphere and NSX UI. -If required for maintenance reasons, it is possible to access the vSphere UI via [wirequard](../docs/wiregard.md) / [jumphost VM](../maintenance-jumphost/README.md). +If required for maintenance reasons, it is possible to access the vSphere UI via [wirequard](../docs/wireguard.md) / [jumphost VM](../maintenance-jumphost/README.md). -After connecting, vSphere UI is available at https://vcsa-427138.d1de5ee9.us-central1.gve.goog. +After connecting, vSphere UI is available at https://vcsa-434004.f63e615b.us-central1.gve.goog . -vSphere credentials are available in the google cloud console, VMware Engine, Private clouds, Detail of the `k8s-gcp-gcve-pc` private cloud, Management Appliances, key details ([link](https://console.cloud.google.com/vmwareengine/privateclouds/us-central1-a/k8s-gcp-gcve-pc/management-appliances?project=broadcom-451918)) +vSphere credentials are available in the google cloud console, VMware Engine, Private clouds, Detail of the `k8s-gcp-gcve` private cloud, Management Appliances, key details ([link](https://console.cloud.google.com/vmwareengine/privateclouds/us-central1-a/k8s-gcp-gcve/management-appliances?project=broadcom-451918)) IMPORTANT: do not apply changes using the vSphere UI, always use terraform, or when not possible scripts in this folder. -Similar considerations apply for NSX, which is avalable at http://nsx-427314.d1de5ee9.us-central1.gve.goog +Similar considerations apply for NSX, which is avalable at http://nsx-434005.f63e615b.us-central1.gve.goog . # Changing the GCVE CI user's password diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/content-library.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/content-library.tf index 0c6542b7bce..5c17253e644 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/content-library.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/content-library.tf @@ -14,6 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Creates the content library used for testing with CAPV. resource "vsphere_content_library" "capv" { name = "capv" description = "Content Library for CAPV." diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/data.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/data.tf index 573edd12637..1f74686ddf0 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/data.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/data.tf @@ -14,6 +14,10 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Read existing information from vSphere. + +# Resources from vSphere. + data "vsphere_datacenter" "datacenter" { name = var.vsphere_datacenter } @@ -28,19 +32,22 @@ data "vsphere_datastore" "datastore" { datacenter_id = data.vsphere_datacenter.datacenter.id } -data "vsphere_role" "read-only" { - label = "Read-only" +data "vsphere_network" "network" { + name = var.vsphere_network_name + datacenter_id = data.vsphere_datacenter.datacenter.id + depends_on = [ nsxt_policy_segment.k8s-ci ] } -data "vsphere_role" "no-access" { - label = "No access" +# Resources from NSX-T. + +data "nsxt_policy_tier1_gateway" "tier1_gw" { + display_name = "Tier1" } -data "vsphere_network" "network" { - name = var.vsphere_network_name - datacenter_id = data.vsphere_datacenter.datacenter.id +data "nsxt_policy_transport_zone" "overlay_tz" { + display_name = "TZ-OVERLAY" } -data "vsphere_folder" "global" { - path = "/" +data "nsxt_policy_edge_cluster" "edge_cluster" { + display_name = "edge-cluster" } \ No newline at end of file diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/folders.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/folders.tf index 6beacc48707..3b7d873f63d 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/folders.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/folders.tf @@ -14,12 +14,14 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Creates the folder prow for hosting all CI workload. resource "vsphere_folder" "prow" { path = "prow" type = "vm" datacenter_id = data.vsphere_datacenter.datacenter.id } +# Creates the folder prow/templates for hosting VM Templates. resource "vsphere_folder" "templates" { path = "${vsphere_folder.prow.path}/templates" type = "vm" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roleassignments.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roleassignments.tf index b1e15981f1a..7c64b7c5c64 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roleassignments.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roleassignments.tf @@ -14,17 +14,9 @@ See the License for the specific language governing permissions and limitations under the License. */ -# Grant read-only access to the datacenter. -# Permissions will later be granted on lower placed resource pools. -## / - -# Deny access recursively to the prow resource pool to restrict the above recursive read-only rule. -# Permissions will later be granted on lower placed resource pools. -## //host//Resources/prow # Grant access on the network. ## //network/ - resource "vsphere_entity_permissions" "permissions_network" { entity_id = data.vsphere_network.network.id entity_type = "Network" @@ -36,12 +28,8 @@ resource "vsphere_entity_permissions" "permissions_network" { } } -# TODO(chrischdi): we propably also have to add ReadOnly to the //network/ - - # Grant read-only access to the templates vm directory. ## //vm/prow/templates - resource "vsphere_entity_permissions" "permissions_templates_directory" { entity_id = vsphere_folder.templates.id entity_type = "ClusterComputeResource" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roles.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roles.tf index b57ed264bb4..a27d3875a3b 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roles.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/iam_roles.tf @@ -14,6 +14,11 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Permission roles required for the CI user. +# Assignment of the roles is done via iam_roleassignments.tf, projects.tf +# and manually via `scripts/ensure-users-permissions.sh`. + +# Creates the vSphere role for the ci user to e.g. create VMs, Resource Pools, Folders, ... during tests. resource "vsphere_role" "vsphere-ci" { name = "vsphere-ci" role_privileges = [ @@ -77,7 +82,7 @@ resource "vsphere_role" "vsphere-ci" { ] } -# allows the ci user to browse CNS and storage profiles. +# Creates the vSphere role "vsphere-ci-readonly" for user access to browse CNS and storage profiles. resource "vsphere_role" "vsphere-ci-readonly" { name = "vsphere-ci-readonly" role_privileges = [ @@ -87,7 +92,7 @@ resource "vsphere_role" "vsphere-ci-readonly" { } -# templates-ci allows users access to the templates folder to clone templates to virtual machines. +# Creates the vSphere role "templates-ci" for user access to the templates folder to clone templates to virtual machines. resource "vsphere_role" "templates-ci" { name = "templates-ci" role_privileges = [ diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/main.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/main.tf index 85438141118..ea797f4b19b 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/main.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/main.tf @@ -14,8 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -# Resource Pools - +# Creates the project specific resource pool resource "vsphere_resource_pool" "project_resource_pool" { count = var.nr_projects @@ -23,8 +22,7 @@ resource "vsphere_resource_pool" "project_resource_pool" { parent_resource_pool_id = var.vsphere_resource_pool_id } -# Folders - +# Creates the project specific folder resource "vsphere_folder" "project_folder" { count = var.nr_projects @@ -33,8 +31,7 @@ resource "vsphere_folder" "project_folder" { datacenter_id = var.vsphere_datacenter_id } -# Permissions - +# Grants permissions to the prow user for the resource pool resource "vsphere_entity_permissions" "permission_resource_pool_project" { count = var.nr_projects @@ -50,6 +47,7 @@ resource "vsphere_entity_permissions" "permission_resource_pool_project" { depends_on = [vsphere_resource_pool.project_resource_pool] } +# Grants permissions to the prow user for the folder resource "vsphere_entity_permissions" "permission_folder_project" { count = var.nr_projects diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/provider.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/provider.tf index 53e840b4fc3..ca5e9a7d987 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/provider.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/provider.tf @@ -15,6 +15,8 @@ limitations under the License. */ terraform { + required_version = "1.10.5" + required_providers { vsphere = { source = "vmware/vsphere" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/variables.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/variables.tf index eac59e741ee..63bb5482abd 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/variables.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/modules/gcp-gcve-project/variables.tf @@ -14,6 +14,8 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Input variables for the module. + variable "project_name" { description = "The name of the project which matches the boskos resource name" type = string diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/nsx-t.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/nsx-t.tf index d44a6596964..faa16d4a801 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/nsx-t.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/nsx-t.tf @@ -14,18 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -data "nsxt_policy_tier1_gateway" "tier1_gw" { - display_name = "Tier1" -} - -data "nsxt_policy_transport_zone" "overlay_tz" { - display_name = "TZ-OVERLAY" -} - -data "nsxt_policy_edge_cluster" "edge_cluster" { - display_name = "edge-cluster" -} - +# Creates a DHCP Server for the workload network. resource "nsxt_policy_dhcp_server" "k8s-ci-dhcp" { display_name = "k8s-ci-dhcp" description = "Terraform provisioned DhcpServerConfig" @@ -34,7 +23,7 @@ resource "nsxt_policy_dhcp_server" "k8s-ci-dhcp" { server_addresses = ["192.168.32.10/21"] } - +# Creates the subnet for hosting the VM workload network. resource "nsxt_policy_segment" "k8s-ci" { display_name = "k8s-ci" connectivity_path = data.nsxt_policy_tier1_gateway.tier1_gw.path @@ -43,12 +32,71 @@ resource "nsxt_policy_segment" "k8s-ci" { subnet { cidr = "192.168.32.1/21" - dhcp_ranges = ["192.168.32.10-192.168.33.255"] + # This is the DHCP range used for created VMs. + # The IP range 192.168.35.0 - 192.168.37.127 is used for VIPs (e.g. via kube-vip) + # and is assigned to boskos projects. + dhcp_ranges = ["192.168.32.11-192.168.33.255"] dhcp_v4_config { server_address = "192.168.32.2/21" dns_servers = [var.gcve_dns_server] lease_time = 600 + + dhcp_generic_option { + # Set correct MTU to avoid package drops. + code = "26" + values = [ "1360" ] + } } } } + +# Static routes to the internet as workaround via the gateway vm at 192.168.32.8. +# This is a workaround for a gcve connectivity issue which limits requests to 64 for a certain amount of time. +# Without this created VMs may e.g. fail during provisioning due to hitting image pull backoffs and thus timeouts. +resource "nsxt_policy_static_route" "route1" { + display_name = "worakround route to ${each.value} via gateway vm" + gateway_path = data.nsxt_policy_tier1_gateway.tier1_gw.path + + network = "${each.value}" + + next_hop { + admin_distance = "1" + ip_address = "192.168.32.8" + interface = nsxt_policy_segment.k8s-ci.path + } + + for_each = toset([ + "0.0.0.0/5", + "8.0.0.0/7", + "11.0.0.0/8", + "12.0.0.0/6", + "16.0.0.0/4", + "32.0.0.0/3", + "64.0.0.0/2", + "128.0.0.0/3", + "160.0.0.0/5", + "168.0.0.0/6", + "172.0.0.0/12", + "172.32.0.0/11", + "172.64.0.0/10", + "172.128.0.0/9", + "173.0.0.0/8", + "174.0.0.0/7", + "176.0.0.0/4", + "192.0.0.0/9", + "192.128.0.0/11", + "192.160.0.0/13", + "192.169.0.0/16", + "192.170.0.0/15", + "192.172.0.0/14", + "192.176.0.0/12", + "192.192.0.0/10", + "193.0.0.0/8", + "194.0.0.0/7", + "196.0.0.0/6", + "200.0.0.0/5", + "208.0.0.0/4", + "224.0.0.0/3", + ]) +} diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/projects.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/projects.tf index 7e473a40f6b..92cb758ec46 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/projects.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/projects.tf @@ -18,13 +18,16 @@ locals { gcp_gcve_project_name = "k8s-infra-e2e-gcp-gcve-project" } -## create the projects (resource pool, folder, assign permissions per resource pool and folder) +# Creates the projects. +# Each project will map to a resources of type `gcve-vsphere-project` in Boskos and provide: +# A resource pool and folder (created by this module including permissions) and +# a IPPool (see [Boskos](../../docs/boskos.md) for more information). module "gcp-gcve-projects" { source = "./modules/gcp-gcve-project" + nr_projects = var.gcp_gcve_nr_projects project_name = local.gcp_gcve_project_name group = var.gcp_gcve_iam_group - nr_projects = var.gcp_gcve_nr_projects role_id = vsphere_role.vsphere-ci.id vsphere_datacenter_id = data.vsphere_datacenter.datacenter.id vsphere_folder_path = vsphere_folder.prow.path diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/provider.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/provider.tf index 7fda4eb7d99..8585eee8073 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/provider.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/provider.tf @@ -14,14 +14,20 @@ See the License for the specific language governing permissions and limitations under the License. */ +/* +This file defines: +- Required provider versions +- Storage backend details +*/ + terraform { + required_version = "1.10.5" backend "gcs" { bucket = "k8s-infra-tf-gcp-gcve" prefix = "k8s-infra-gcp-gcve-vcenter" } - required_providers { google = { source = "hashicorp/google" @@ -42,12 +48,14 @@ terraform { } } +# Setup credentials to vSphere. provider "vsphere" { user = var.vsphere_user password = var.vsphere_password vsphere_server = var.vsphere_server } +# Setup credentials to NSX-T. provider "nsxt" { host = var.nsxt_server username = var.nsxt_user diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/resource-pools.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/resource-pools.tf index e63998e26db..dd0e4fd11d1 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/resource-pools.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/resource-pools.tf @@ -14,12 +14,14 @@ See the License for the specific language governing permissions and limitations under the License. */ +# Creates the resource pool prow for hosting all CI workload. resource "vsphere_resource_pool" "prow" { name = "prow" parent_resource_pool_id = data.vsphere_compute_cluster.compute_cluster.resource_pool_id scale_descendants_shares = "disabled" } +# Creates the resource pool prow/templates for hosting VM Templates. resource "vsphere_resource_pool" "templates" { name = "templates" parent_resource_pool_id = vsphere_resource_pool.prow.id diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/boskos-userdata.sh b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/boskos-userdata.sh index b27a4297558..90f514689f2 100755 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/boskos-userdata.sh +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/boskos-userdata.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# The network used is 192.168.0.32/21 +# The network used is 192.168.32.0/21 # Usable Host IP Range: 192.168.32.1 - 192.168.39.254: # DHCP Range: 192.168.32.10 - 192.168.33.255 # Used IPPool for 40 Projects having 16 IPs each: 192.168.35.0 - 192.168.37.127 @@ -44,15 +44,22 @@ function initBoskosResourceUserData() { # shellcheck disable=SC2089 ipPool="{\\\"addresses\\\":[\\\"${START}-${END}\\\"],\\\"gateway\\\":\\\"192.168.32.1\\\",\\\"prefix\\\":21}" + boskos_data='{"ipPool":"'"${ipPool}"'","resourcePool":"'"${resourcePool}"'","folder":"'"${folder}"'"}' + # acquire from "dirty" or "free" state curl -s -X POST "${BOSKOS_HOST}/acquirebystate?names=${resourceName}&state=dirty&dest=busy&owner=$(whoami)" | grep -q "${resourceName}" \ || curl -s -X POST "${BOSKOS_HOST}/acquirebystate?names=${resourceName}&state=free&dest=busy&owner=$(whoami)" | grep -q "${resourceName}" \ || echo "Failed to acquire ${resourceName}" - # update + + # update resource + echo "Updating resource ${resourceName} with following data: ${boskos_data}" # shellcheck disable=SC2089 - curl -s -X POST -d '{"ipPool":"'"${ipPool}"'","resourcePool":"'"${resourcePool}"'","folder":"'"${folder}"'"}' "${BOSKOS_HOST}/update?name=${resourceName}&state=busy&owner=$(whoami)" + curl -s -X POST -d "${boskos_data}" "${BOSKOS_HOST}/update?name=${resourceName}&state=busy&owner=$(whoami)" + # release as "dirty", janitor should bring it to "free" - curl -s -X POST "${BOSKOS_HOST}/release?name=${resourceName}&dest=dirty&owner=$(whoami)" + curl -s -X POST "${BOSKOS_HOST}/release?name=${resourceName}&dest=dirty&owner=$(whoami)" + + echo "Successfully updated project ${resourceName}" } for i in {1..40}; do diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-essential-contacts.sh b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-essential-contacts.sh new file mode 100755 index 00000000000..94d2816031a --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-essential-contacts.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# Copyright 2025 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Adds all users from the group k8s-infra-gcp-gcve-admins to the essential contacts of the project. +# This allows receiving notifications for updates, etc. for VMware Engine. +# xref: https://cloud.google.com/vmware-engine/docs/concepts-monitoring#email-notifications + +REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../../.." && pwd -P) +readonly REPO_ROOT + +GCLOUD_PROJECT_ID=broadcom-451918 + +for mail in $( < "${REPO_ROOT}/groups/sig-k8s-infra/groups.yaml" yq -r '.groups[] | select(.name == "k8s-infra-gcp-gcve-admins") | .members[] | select(. | contains "@kubernetes.io" | not)'); do + echo "> Ensuring ${mail} exists as essential contact" + ENTRIES="$(gcloud essential-contacts list --filter "email=${mail}" --format=json | jq '. | length')" + if [[ $ENTRIES -eq 0 ]]; then + echo "Creating ${mail} as technical essential contact" + gcloud essential-contacts create --language=en-US --notification-categories=legal,product-updates,security,suspension,technical --project "${GCLOUD_PROJECT_ID}" "--email=${mail}" + elif [[ $ENTRIES -eq 1 ]]; then + echo "Contact already exists" + else + echo "ERROR: checking ${mail}, too many results" + exit 1 + fi +done diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-users-permissions.sh b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-users-permissions.sh index 38a7fe59956..2c7afca86ba 100755 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-users-permissions.sh +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/ensure-users-permissions.sh @@ -25,6 +25,9 @@ govc permissions.ls / | grep ${GCVE_PROW_CI_GROUP} | grep -q vsphere-ci-readonly # Enable searching for VM's and top-level read permissions. govc permissions.ls /Datacenter | grep ${GCVE_PROW_CI_GROUP} | grep -q ReadOnly || (govc permissions.set -propagate=false -group -principal GVE.LOCAL\\${GCVE_PROW_CI_GROUP} -role ReadOnly /Datacenter && echo "Added permissions for ${GCVE_PROW_CI_GROUP} as ReadOnly to /Datacenter") +# Enable searching for ComputeCluster +govc permissions.ls /Datacenter/host/k8s-gcve-cluster | grep ${GCVE_PROW_CI_GROUP} | grep -q ReadOnly || (govc permissions.set -propagate=false -group -principal GVE.LOCAL\\${GCVE_PROW_CI_GROUP} -role ReadOnly /Datacenter/host/k8s-gcve-cluster && echo "Added permissions for ${GCVE_PROW_CI_GROUP} as ReadOnly to /Datacenter/host/k8s-gcve-cluster") + # Enable read for DistributedVirtualSwitch govc permissions.ls /Datacenter/network/Datacenter-dvs | grep ${GCVE_PROW_CI_GROUP} | grep -q ReadOnly || (govc permissions.set -propagate=false -group -principal GVE.LOCAL\\${GCVE_PROW_CI_GROUP} -role ReadOnly /Datacenter/network/Datacenter-dvs && echo "Added permissions for ${GCVE_PROW_CI_GROUP} as ReadOnly to /Datacenter/network/Datacenter-dvs") @@ -33,3 +36,8 @@ govc permissions.ls /Datacenter/datastore/vsanDatastore | grep ${GCVE_PROW_CI_GR # Allow access to content libraries govc sso.group.ls "RegistryAdministrators" | grep -q "${GCVE_PROW_CI_GROUP}" || (govc sso.group.update -g -a "${GCVE_PROW_CI_GROUP}" "RegistryAdministrators" && echo "Added group ${GCVE_PROW_CI_GROUP} to group RegistryAdministrators") + +# Attach tags for failure domain testing +# The tags are created via ../tags.tf +govc tags.attached.ls -l Datacenter | grep -q Datacenter || (govc tags.attach -c k8s-region Datacenter /Datacenter && echo "Added tag Datacenter of category k8s-region to Datacenter") +govc tags.attached.ls -l k8s-gcve-cluster | grep -q k8s-gcve-cluster || (govc tags.attach -c k8s-zone k8s-gcve-cluster /Datacenter/host/k8s-gcve-cluster && echo "Added tag k8s-gcve-cluster of category k8s-zone to k8s-gcve-cluster") diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/upload-ova.sh b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/upload-ova.sh new file mode 100755 index 00000000000..fb0420decfc --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/scripts/upload-ova.sh @@ -0,0 +1,80 @@ +#!/bin/bash + +# Copyright 2025 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +GOVC_FOLDER="/Datacenter/vm/prow/templates" +OVC_DATASTORE="vsanDatastore" +VCENTER_NETWORK_NAME=k8s-ci +export GOVC_FOLDER +export OVC_DATASTORE +export VCENTER_NETWORK_NAME + +function checkIfExists() { + OVA_TEMPLATE_NAME="${1}" + if govc ls "${GOVC_FOLDER}/${OVA_TEMPLATE_NAME}" | grep -q "$OVA_TEMPLATE_NAME$"; then + echo "ERROR: OVA already exists at ${GOVC_FOLDER}/${OVA_TEMPLATE_NAME}" + return 1 + fi +} + +# downloadAndConcatenate downloads an ova from a url. If the url is split in parts, multiple urls should be given in the right order. +# It also tries to get the sha256 checksum from the same url. +# It assumes multi-part urls are named `*.ova-part-[a-z]+` and the sha256 sum file is named `.*.ova.sha256` +function downloadAndConcatenate() { + NAME="$(echo "${1}" | tr '/' ' ' | awk '{print $NF}' | sed 's/\.ova.*/\.ova/')" + SHA256SUM="$(curl -L -s "$(echo "${1}" | awk '{print $1}' | sed -E -e 's/\.ova(|-part-[a-z]+).*/.ova.sha256/')")" + # shellcheck disable=SC2068 + for dl in ${@}; do + echo "Downloading ${NAME} via $dl" + curl -L "$dl" >> "${NAME}" + done + echo "${SHA256SUM} ${NAME}" | sha256sum --check --status || echo "ERROR: sha256sum for ${NAME} is invalid!" +} + +# importOVA imports an ova to vCenter as template (including a snapshot for linked clones) and to the content library +function importOVA() { + OVA_PATH="$1" + OVA_NAME=${OVA_PATH##*/} + OVA_TEMPLATE_NAME=${OVA_NAME%.ova} + + echo "Importing ${OVA_NAME} from ${OVA_PATH}" + + govc import.spec "${OVA_PATH}" | jq '.NetworkMapping[] |= (.Network="'$VCENTER_NETWORK_NAME'")' > ova_spec.json + echo "# uploading OVA to VM" + govc import.ova "-name=${OVA_TEMPLATE_NAME}" -options=ova_spec.json "${OVA_PATH}" + echo "# Creating a snapshot for linked clone" + govc snapshot.create -vm "${OVA_TEMPLATE_NAME}" for-linkedclone + echo "# Marking as VM as Template" + govc vm.markastemplate "${OVA_TEMPLATE_NAME}" + echo "# Import OVA to content library /capv" + govc library.import /capv "${OVA_PATH}" +} + +if [ $# -lt 1 ]; then + echo "ERROR: Expecting url(s) to an OVA as arguments." + echo "Example usage:" + echo " ${0} https://url-to.ova" + echo " ${0} https://url-to.ova-part-aa https://url-to.ova-part-ab" + exit 1 +fi + +OVA_PATH="$(echo "${1}" | tr '/' ' ' | awk '{print $NF}' | sed 's/\.ova.*/\.ova/')" +OVA_NAME=${OVA_PATH##*/} +OVA_TEMPLATE_NAME=${OVA_NAME%.ova} + +checkIfExists "${OVA_TEMPLATE_NAME}" || exit 1 +# shellcheck disable=SC2068 +downloadAndConcatenate $@ +importOVA "${OVA_PATH}" diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/tags.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/tags.tf new file mode 100644 index 00000000000..b7862331036 --- /dev/null +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/tags.tf @@ -0,0 +1,51 @@ +/* +Copyright 2025 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +# Creates the tag category "k8s-region" required for failure domain testing. +resource "vsphere_tag_category" "category_k8s_region" { + name = "k8s-region" + cardinality = "MULTIPLE" + description = "Managed by Terraform" + + associable_types = [ + "Datacenter", + ] +} + +# Creates the tag category "k8s-zone" required for failure domain testing. +resource "vsphere_tag_category" "category_k8s_zone" { + name = "k8s-zone" + cardinality = "MULTIPLE" + description = "Managed by Terraform" + + associable_types = [ + "ClusterComputeResource", + ] +} + +# Creates the Datacenter tag for the k8s-region tag category for failure domain testing. +resource "vsphere_tag" "tag_k8s_region" { + name = var.vsphere_datacenter + category_id = vsphere_tag_category.category_k8s_region.id + description = "Managed by Terraform" +} + +# Creates the Compute Cluster tag for the k8s-zone tag category for failure domain testing. +resource "vsphere_tag" "tag_k8s_zone" { + name = var.vsphere_cluster + category_id = vsphere_tag_category.category_k8s_zone.id + description = "Managed by Terraform" +} diff --git a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/variables.tf b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/variables.tf index 6eac1079e4d..e06d7435d2b 100644 --- a/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/variables.tf +++ b/infra/gcp/terraform/k8s-infra-gcp-gcve/vsphere/variables.tf @@ -14,6 +14,8 @@ See the License for the specific language governing permissions and limitations under the License. */ +# solution admin user from GCVE +# xref: https://cloud.google.com/vmware-engine/docs/private-clouds/howto-elevate-privilege variable "vsphere_user" { type = string } @@ -25,52 +27,60 @@ variable "vsphere_password" { variable "vsphere_server" { type = string } - -variable "nsxt_server" { +# NSX-T user from GCVE dashboard +# xref: http://console.cloud.google.com/vmwareengine/privateclouds/us-central1-a/k8s-gcp-gcve/management-appliances?project=broadcom-451918 +variable "nsxt_user" { type = string } -variable "nsxt_user" { +variable "nsxt_password" { type = string } -variable "nsxt_password" { + +variable "nsxt_server" { type = string } -# This DNS Server was created via GCVE and can be found in GCVE's summary page. +# This DNS Server was created by GCVE and can be found in GCVE's summary page. +# xref: https://console.cloud.google.com/vmwareengine/privateclouds/us-central1-a/k8s-gcp-gcve/management-appliances?project=broadcom-451918 variable "gcve_dns_server" { type = string - default = "192.168.30.234" + default = "192.168.31.234" } +# This is the name of the Datacenter created by GCVE variable "vsphere_datacenter" { type = string default = "Datacenter" } +# This is the name of the VMware Engine Private Cloud created via ../vmware-engine.tf variable "vsphere_cluster" { type = string default = "k8s-gcve-cluster" } +# This is the name of the Datastore created by GCVE variable "vsphere_datastorename" { type = string default = "vsanDatastore" } +# This is the name of the Network which will be created in NSX-T variable "vsphere_network_name" { type = string default = "k8s-ci" } -# Variables specific to cluster-api-provider-vsphere - +# This is the name of the IAM group specific to prow CI and created via scripts/ensure-users-groups.sh variable "gcp_gcve_iam_group" { type = string default = "GVE.LOCAL\\prow-ci-group" } +# This is the number of projects to be created in vSphere. +# It should match the number of boskos projects. variable "gcp_gcve_nr_projects" { type = number default = 40