Skip to content

Commit 7d865f1

Browse files
winlinuxmattwinlinuxmatt
authored
Merge pull request #17 from winlinuxmatt/mbarnes/update_cluster_parms
update code to be clean
2 parents 3362618 + 25ac09d commit 7d865f1

File tree

7 files changed

+118
-178
lines changed

7 files changed

+118
-178
lines changed

‎README.md‎

Lines changed: 55 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -1,117 +1,81 @@
1-
# kubernetes_iac
2-
Terraform Kubernetes IAC
1+
# Kubernetes IAC
32

4-
This repository was constructed by following the instructions outlined in the enlightening article mentioned below. I have made certain modifications to adapt it to my specific cluster configuration. It is worth noting that the datasets or datastores on my proxmox might have different names compared to those in your proxmox setup. Therefore, it is essential to make the necessary adjustments according to your own setup. My proxmox cluster consists of three nodes and utilizes the ceph storage solution for efficient management of virtual machines across all nodes.
3+
Terraform Kubernetes Infrastructure as Code (IAC)
54

5+
This repository was created by following the instructions in the article linked below, with modifications to suit my specific cluster configuration. Note that the datasets or datastores on my Proxmox setup may differ from yours, so adjust accordingly. My Proxmox cluster consists of three nodes and uses Ceph for efficient virtual machine management across all nodes.
66

7-
Article: [Talos Cluster on Proxmox with Terraform](https://olav.ninja/talos-cluster-on-proxmox-with-terraform) by Olav
7+
**Article**: [Talos Cluster on Proxmox with Terraform](https://olav.ninja/talos-cluster-on-proxmox-with-terraform) by Olav
8+
9+
---
810

911
## Additional Steps
1012

11-
After setting up the cluster, there are a few additional steps that may be helpful to others.
13+
After setting up the cluster, you may find the following steps helpful.
1214

1315
### Connect to the Talos Kubernetes Cluster
1416

15-
To connect to your Talos Kubernetes cluster using the outputs from Terraform, you need to set up your local environment with the correct configuration files. Follow these steps:
16-
17-
1. Save the `kubeconfig` and `talosconfig` outputs to files on your local machine. Run the following commands:
17+
To connect to your Talos Kubernetes cluster using Terraform outputs, configure your local environment as follows:
1818

19+
1. Save the `kubeconfig` and `talosconfig` outputs to files on your local machine:
1920
```bash
20-
terraform output -raw kubeconfig > ~/.kube/config && terraform output -raw talosconfig > ~/.talos/config
21+
terraform output -raw kubeconfig > ~/.kube/config
22+
terraform output -raw talosconfig > ~/.talos/config
2123
```
2224

23-
Note: Adjust the file paths if necessary.
24-
25-
2. Verify the file permissions for these files to avoid any security issues:
26-
25+
2. Set appropriate file permissions to avoid security issues:
2726
```bash
28-
chmod 600 ~/.kube/config && chmod 600 ~/.talos/config
27+
chmod 600 ~/.kube/config ~/.talos/config
2928
```
3029

31-
### Set Up kubectl
32-
33-
To interact with the Kubernetes cluster, you can use `kubectl`. Here's an example command to get the list of nodes:
34-
35-
```
36-
terraform output -raw kubeconfig > ~/.kube/config && terraform output -raw talosconfig > ~/.talos/config
37-
```
38-
39-
This assumes you have ~/.kube/config and ~/.talos/config as the default locations for Kubernetes and Talos configurations. Adjust the file paths if necessary.
40-
Verify File Permissions
41-
Ensure the permissions for these files are set correctly to avoid any security issues:
42-
43-
```
44-
chmod 600 ~/.kube/config && chmod 600 ~/.talos/config
45-
```
46-
47-
Or you can do it all in one
30+
### Set Up `kubectl`
4831

49-
```
50-
terraform output -raw kubeconfig > ~/.kube/config && terraform output -raw talosconfig > ~/.talos/config && chmod 600 ~/.kube/config && chmod 600 ~/.talos/config
51-
```
52-
53-
Set Up kubectl
54-
Get the list of nodes
55-
56-
```
32+
To interact with the Kubernetes cluster, use `kubectl`. For example, to list the nodes:
33+
```bash
5734
kubectl get nodes
5835
```
5936

37+
Sample output:
6038
```
61-
NAME STATUS ROLES AGE VERSION
62-
talos-cp-01 Ready control-plane 3h39m v1.30.0
63-
talos-cp-02 Ready control-plane 3h24m v1.30.0
64-
talos-cp-03 Ready control-plane 3h24m v1.30.0
65-
talos-worker-01 Ready <none> 3h39m v1.30.0
66-
talos-worker-02 Ready <none> 3h24m v1.30.0
67-
talos-worker-03 Ready <none> 3h24m v1.30.0
68-
```
69-
Show the dashboard of one of the nodes via
70-
71-
```
72-
talosctl dashboard -n talos-cp-01
39+
NAME STATUS ROLES AGE VERSION
40+
talos-cp-01 Ready control-plane 83s v1.32.0
41+
talos-cp-02 Ready control-plane 86s v1.32.0
42+
talos-cp-03 Ready control-plane 85s v1.32.0
43+
talos-worker-01 Ready <none> 88s v1.32.0
44+
talos-worker-02 Ready <none> 86s v1.32.0
45+
talos-worker-03 Ready <none> 90s v1.32.0
7346
```
74-
Check the Health
7547

76-
```
77-
talosctl -n talos-cp-01 health 1 ↵
78-
discovered nodes: ["10.0.0.73" "10.0.0.74" "10.0.0.75" "10.0.0.70" "10.0.0.71" "10.0.0.72"]
79-
waiting for etcd to be healthy: ...
80-
waiting for etcd to be healthy: OK
81-
waiting for etcd members to be consistent across nodes: ...
82-
waiting for etcd members to be consistent across nodes: OK
83-
waiting for etcd members to be control plane nodes: ...
84-
waiting for etcd members to be control plane nodes: OK
85-
waiting for apid to be ready: ...
86-
waiting for apid to be ready: OK
87-
waiting for all nodes memory sizes: ...
88-
waiting for all nodes memory sizes: OK
89-
waiting for all nodes disk sizes: ...
90-
waiting for all nodes disk sizes: OK
91-
waiting for kubelet to be healthy: ...
92-
waiting for kubelet to be healthy: OK
93-
waiting for all nodes to finish boot sequence: ...
94-
waiting for all nodes to finish boot sequence: OK
95-
waiting for all k8s nodes to report: ...
96-
waiting for all k8s nodes to report: OK
97-
waiting for all k8s nodes to report ready: ...
98-
waiting for all k8s nodes to report ready: OK
99-
waiting for all control plane static pods to be running: ...
100-
waiting for all control plane static pods to be running: OK
101-
waiting for all control plane components to be ready: ...
102-
waiting for all control plane components to be ready: OK
103-
waiting for kube-proxy to report ready: ...
104-
waiting for kube-proxy to report ready: OK
105-
waiting for coredns to report ready: ...
106-
waiting for coredns to report ready: OK
107-
waiting for all k8s nodes to report schedulable: ...
108-
waiting for all k8s nodes to report schedulable: OK
48+
### Use `talosctl`
49+
50+
- **View the Dashboard**:
51+
```bash
52+
talosctl dashboard -n talos-cp-01
53+
```
54+
55+
- **Check Cluster Health**:
56+
```bash
57+
talosctl -n talos-cp-01 health
58+
```
59+
Sample output:
60+
```
61+
discovered nodes: ["10.0.0.73" "10.0.0.74" "10.0.0.75" "10.0.0.70" "10.0.0.71" "10.0.0.72"]
62+
waiting for etcd to be healthy: OK
63+
waiting for all k8s nodes to report ready: OK
64+
waiting for all control plane components to be ready: OK
65+
...
66+
```
67+
68+
- **Health Dashboard Example**:
69+
![Talosctl Dashboard](photos/talosctl_dashboard.png)
70+
71+
### Reset the Cluster
72+
73+
If you need to start over, you can taint resources and reapply the Terraform configuration:
74+
```bash
75+
terraform state list | xargs -n1 terraform taint
76+
terraform apply
10977
```
11078

111-
Check the health of the node/nodes
112-
![](photos/talosctl_dashboard.png)
79+
---
11380

114-
If you need to start over you can always taint things and terraform apply again
115-
```
116-
terraform state list | xargs -n1 terraform taint
117-
```
81+
Adjust paths and configurations as needed for your environment.

‎cluster.tf‎

Lines changed: 32 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -1,131 +1,113 @@
1-
# This is how the whole setup is structured:
2-
# - **main.tf**: This file contains the main configuration for managing the Talos cluster on Proxmox, including provider definitions, resource configurations, and data blocks for retrieving machine configurations.
3-
# - **providers.tf**: This file defines the required providers for the Terraform configuration, specifying the source and version for each provider.
4-
# - **variables.tf**: This file defines the variables used in the Terraform configuration, allowing for customization of the cluster name, IP addresses, and other settings.
5-
# - **virtual_machines.tf**: This file defines the virtual machines to be created in the Proxmox virtual environment, specifying their configurations such as CPU, memory, disk, and network settings.
6-
# - **cluster.tf**: This file contains the configuration for the Talos cluster, including the creation of machine secrets, retrieval of machine configurations, application of configurations to control plane and worker nodes, bootstrapping the control plane, and retrieving the health status and kubeconfig for the cluster.
7-
# This resource creates a new Talos machine secrets resource.
1+
# Talos Cluster Configuration on Proxmox
2+
3+
# Talos Machine Secrets
84
resource "talos_machine_secrets" "machine_secrets" {}
95

10-
# This data block retrieves the client configuration for the Talos cluster.
6+
# Talos Client Configuration
117
data "talos_client_configuration" "talosconfig" {
128
cluster_name = var.cluster_name
139
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
1410
endpoints = [var.talos_cp_01_ip_addr]
1511
}
1612

17-
# This data block retrieves the machine configuration for the "cp_01" node from the Talos provider.
13+
# Control Plane Machine Configurations
1814
data "talos_machine_configuration" "machineconfig_cp" {
1915
cluster_name = var.cluster_name
2016
cluster_endpoint = "https://${var.talos_cp_01_ip_addr}:6443"
2117
machine_type = "controlplane"
2218
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
2319
}
2420

25-
# This data block retrieves the machine configuration for the "cp_02" node from the Talos provider.
2621
data "talos_machine_configuration" "machineconfig_cp_02" {
2722
cluster_name = var.cluster_name
2823
cluster_endpoint = "https://${var.talos_cp_02_ip_addr}:6443"
2924
machine_type = "controlplane"
3025
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
3126
}
3227

33-
# This data block retrieves the machine configuration for the "cp_03" node from the Talos provider.
3428
data "talos_machine_configuration" "machineconfig_cp_03" {
3529
cluster_name = var.cluster_name
3630
cluster_endpoint = "https://${var.talos_cp_03_ip_addr}:6443"
3731
machine_type = "controlplane"
3832
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
3933
}
4034

41-
# This resource applies the machine configuration for the "cp_01" node.
35+
# Apply Control Plane Configurations
4236
resource "talos_machine_configuration_apply" "cp_config_apply" {
43-
depends_on = [ proxmox_virtual_environment_vm.talos_cp_01 ]
37+
depends_on = [proxmox_virtual_environment_vm.talos_cp_01]
4438
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
4539
machine_configuration_input = data.talos_machine_configuration.machineconfig_cp.machine_configuration
46-
count = 1
4740
node = var.talos_cp_01_ip_addr
4841
}
4942

50-
# This resource applies the machine configuration for the "cp_02" node.
5143
resource "talos_machine_configuration_apply" "cp_config_apply_02" {
52-
depends_on = [ proxmox_virtual_environment_vm.talos_cp_02 ]
44+
depends_on = [proxmox_virtual_environment_vm.talos_cp_02]
5345
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
5446
machine_configuration_input = data.talos_machine_configuration.machineconfig_cp_02.machine_configuration
55-
count = 1
5647
node = var.talos_cp_02_ip_addr
5748
}
5849

59-
# This resource applies the machine configuration for the "cp_03" node.
6050
resource "talos_machine_configuration_apply" "cp_config_apply_03" {
61-
depends_on = [ proxmox_virtual_environment_vm.talos_cp_03 ]
51+
depends_on = [proxmox_virtual_environment_vm.talos_cp_03]
6252
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
6353
machine_configuration_input = data.talos_machine_configuration.machineconfig_cp_03.machine_configuration
64-
count = 1
6554
node = var.talos_cp_03_ip_addr
6655
}
6756

68-
# This data block retrieves the machine configuration for the "worker" nodes from the Talos provider.
57+
# Worker Machine Configurations
6958
data "talos_machine_configuration" "machineconfig_worker" {
7059
cluster_name = var.cluster_name
7160
cluster_endpoint = "https://${var.talos_cp_01_ip_addr}:6443"
7261
machine_type = "worker"
7362
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
7463
}
7564

76-
# This data block retrieves the machine configuration for the "worker_02" node from the Talos provider.
7765
data "talos_machine_configuration" "machineconfig_worker_02" {
7866
cluster_name = var.cluster_name
7967
cluster_endpoint = "https://${var.talos_cp_02_ip_addr}:6443"
8068
machine_type = "worker"
8169
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
8270
}
8371

84-
# This data block retrieves the machine configuration for the "worker_03" node from the Talos provider.
8572
data "talos_machine_configuration" "machineconfig_worker_03" {
8673
cluster_name = var.cluster_name
8774
cluster_endpoint = "https://${var.talos_cp_03_ip_addr}:6443"
8875
machine_type = "worker"
8976
machine_secrets = talos_machine_secrets.machine_secrets.machine_secrets
9077
}
9178

92-
# This resource applies the machine configuration for the "worker_01" node.
79+
# Apply Worker Configurations
9380
resource "talos_machine_configuration_apply" "worker_config_apply" {
94-
depends_on = [ proxmox_virtual_environment_vm.talos_worker_01 ]
81+
depends_on = [proxmox_virtual_environment_vm.talos_worker_01]
9582
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
9683
machine_configuration_input = data.talos_machine_configuration.machineconfig_worker.machine_configuration
97-
count = 1
9884
node = var.talos_worker_01_ip_addr
9985
}
10086

101-
# This resource applies the machine configuration for the "worker_02" node.
10287
resource "talos_machine_configuration_apply" "worker_config_apply_02" {
103-
depends_on = [ proxmox_virtual_environment_vm.talos_worker_02 ]
88+
depends_on = [proxmox_virtual_environment_vm.talos_worker_02]
10489
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
10590
machine_configuration_input = data.talos_machine_configuration.machineconfig_worker_02.machine_configuration
106-
count = 1
10791
node = var.talos_worker_02_ip_addr
10892
}
10993

110-
# This resource applies the machine configuration for the "worker_03" node.
11194
resource "talos_machine_configuration_apply" "worker_config_apply_03" {
112-
depends_on = [ proxmox_virtual_environment_vm.talos_worker_03 ]
95+
depends_on = [proxmox_virtual_environment_vm.talos_worker_03]
11396
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
11497
machine_configuration_input = data.talos_machine_configuration.machineconfig_worker_03.machine_configuration
115-
count = 1
11698
node = var.talos_worker_03_ip_addr
11799
}
118100

119-
# This resource bootstraps the Talos control plane node.
101+
# Bootstrap Control Plane
120102
resource "talos_machine_bootstrap" "bootstrap" {
121-
depends_on = [ talos_machine_configuration_apply.cp_config_apply ]
103+
depends_on = [talos_machine_configuration_apply.cp_config_apply]
122104
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
123105
node = var.talos_cp_01_ip_addr
124106
}
125107

126-
# This data block retrieves the health status of the Talos cluster.
108+
# Cluster Health Check
127109
data "talos_cluster_health" "health" {
128-
depends_on = [
110+
depends_on = [
129111
talos_machine_configuration_apply.cp_config_apply,
130112
talos_machine_configuration_apply.cp_config_apply_02,
131113
talos_machine_configuration_apply.cp_config_apply_03,
@@ -147,61 +129,44 @@ data "talos_cluster_health" "health" {
147129
endpoints = data.talos_client_configuration.talosconfig.endpoints
148130
}
149131

150-
# This data block retrieves the kubeconfig for the Talos cluster.
132+
# Retrieve Kubeconfig
151133
resource "talos_cluster_kubeconfig" "kubeconfig" {
152-
depends_on = [ talos_machine_bootstrap.bootstrap, data.talos_cluster_health.health ]
134+
depends_on = [talos_machine_bootstrap.bootstrap, data.talos_cluster_health.health]
153135
client_configuration = talos_machine_secrets.machine_secrets.client_configuration
154136
node = var.talos_cp_01_ip_addr
155137
}
156138

157-
# Output the Talos configuration and kubeconfig.
139+
# Outputs
158140
output "talosconfig" {
159141
value = data.talos_client_configuration.talosconfig.talos_config
160142
sensitive = true
161143
}
162144

163-
# Output the kubeconfig for the Talos cluster.
164145
output "kubeconfig" {
165146
value = talos_cluster_kubeconfig.kubeconfig.kubeconfig_raw
166147
sensitive = true
167148
}
168149

169-
# Run custom script for further configuration.
150+
# Custom Script for Configuration
170151
resource "null_resource" "run_custom_script" {
171152
provisioner "local-exec" {
172-
command = "mkdir -p ~/.kube && mkdir -p ~/.talos && terraform output -raw kubeconfig > ~/.kube/config && terraform output -raw talosconfig > ~/.talos/config && chmod 600 ~/.kube/config ~/.talos/config"
153+
command = <<EOT
154+
mkdir -p ~/.kube ~/.talos
155+
terraform output -raw kubeconfig > ~/.kube/config
156+
terraform output -raw talosconfig > ~/.talos/config
157+
chmod 600 ~/.kube/config ~/.talos/config
158+
EOT
173159
}
160+
174161
triggers = {
175-
kubeconfig = talos_cluster_kubeconfig.kubeconfig.kubeconfig_raw
162+
kubeconfig = talos_cluster_kubeconfig.kubeconfig.kubeconfig_raw
176163
talosconfig = data.talos_client_configuration.talosconfig.talos_config
177-
timestamp = timestamp() # Ensure the resource always detects changes
164+
timestamp = timestamp() # Ensure the resource always detects changes
178165
}
166+
179167
depends_on = [
180168
talos_cluster_kubeconfig.kubeconfig,
181169
data.talos_client_configuration.talosconfig,
182170
data.talos_cluster_health.health
183171
]
184172
}
185-
186-
# # Run custom script for further configuration.
187-
# resource "null_resource" "run_custom_script" {
188-
# provisioner "local-exec" {
189-
# command = <<EOT
190-
# mkdir -p ~/.kube && mkdir -p ~/.talos
191-
# terraform output -raw kubeconfig > ~/.kube/config
192-
# terraform output -raw talosconfig > ~/.talos/config
193-
# chmod 600 ~/.kube/config ~/.talos/config
194-
# EOT
195-
# }
196-
# triggers = {
197-
# kubeconfig = try(talos_cluster_kubeconfig.kubeconfig.kubeconfig_raw, "")
198-
# talosconfig = try(data.talos_client_configuration.talosconfig.talos_config, "")
199-
# }
200-
# depends_on = [
201-
# talos_cluster_kubeconfig.kubeconfig,
202-
# data.talos_client_configuration.talosconfig,
203-
# data.talos_cluster_health.health
204-
# ]
205-
# }
206-
207-

0 commit comments

Comments
 (0)