Enterprise-Grade Kubernetes (K3s) Bare-Metal Cloud Architecture

Project Overview

This repository documents the architecture, deployment, and configuration of a highly available, decoupled container orchestration platform built entirely from scratch on bare metal.

graph TD
    %% Styling
    classDef client fill:#d4edda,stroke:#28a745,stroke-width:2px,color:black;
    classDef network fill:#cce5ff,stroke:#007bff,stroke-width:2px,color:black;
    classDef compute fill:#fff3cd,stroke:#ffc107,stroke-width:2px,color:black;
    classDef storage fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:black;

    User([👤 User Browser]):::client -->|HTTP/S Request| DNS{Local DNS}:::client
    
    subgraph Traffic Management [Networking - AWS ALB/Route53 Equivalent]
        DNS -->|Resolves Domain| Traefik[Traefik Ingress Controller]:::network
    end

    subgraph Bare Metal Hypervisor [Compute - AWS EC2/EKS Equivalent]
        Traefik -->|Routes Traffic| Worker1[K3s Worker Node <br/> Nginx Pod]:::compute
        
        Master[K3s Master Node <br/> Tainted Control Plane]:::compute -.->|Manages Cluster| Worker1
    end

    subgraph TrueNAS Core [Storage - AWS EFS Equivalent]
        ZFS[(ZFS Storage Pool)]:::storage -->|NFS Subdir Provisioner| PVC[Dynamic PVCs]:::storage
    end

    Worker1 <-->|Mounts Persistent Volume| PVC

Loading

The objective was to move beyond standard virtual machines to engineer a production-ready cloud environment. By managing the infrastructure at the OS, networking, and orchestration layers, this project implements strict separation of state and compute, dynamic volume provisioning, and automated reverse proxy routing.

Architecture & AWS Enterprise Equivalents

This on-premise infrastructure was specifically designed to mirror the enterprise cloud architectures heavily tested in the AWS Solutions Architect ecosystem.

1. Compute Layer (AWS EC2 Equivalent)

• Hypervisor: Proxmox VE
• Compute Nodes: Ubuntu 24.04 LTS Virtual Machines.
• Implementation: Provisioned dedicated master and worker nodes, dynamically resizing OS partitions and managing Linux file systems (ext4/LVM) to ensure adequate container runtime capacity.

2. Orchestration Layer (AWS EKS Equivalent)

• Engine: K3s (Lightweight Kubernetes)
• Implementation: Bootstrapped a multi-node cluster. To enforce enterprise security, Taints (NoSchedule) were applied to the Control Plane (Master Node) to strictly isolate management workloads from compute workloads, mirroring the AWS EKS Control Plane isolation.

3. Storage Fabric (AWS EFS Equivalent)

• Storage Backend: TrueNAS (ZFS Pool) via NFS.
• Provisioner: NFS Subdir External Provisioner (Deployed via Helm).
• Implementation: Decoupled container state from the Proxmox compute nodes. Pods dynamically request storage via PersistentVolumeClaims (PVCs). The cluster automatically creates the dataset directory on TrueNAS and mounts it over the network. If a compute node fails, workloads are rescheduled and instantly reconnect to their persistent datasets.

4. Traffic Management (AWS ALB & Route 53 Equivalent)

• Ingress Controller: Traefik (Reverse Proxy)
• Implementation: Replaced manual NodePort mapping with automated Ingress routes. Configured Traefik to read incoming HTTP/HTTPS requests and route them to the correct internal cluster IP based on the Host header. Local DNS overrides were utilized to simulate A-Record domain resolution (e.g., routing app.homelab.local to backend pods).

---

Technical Challenges & Resolutions

Building infrastructure from the operating system up requires deep systems troubleshooting. Key challenges resolved during this build include:

• Authentication & Formatting Failures: The initial Worker node join sequence hung indefinitely. By bypassing the system manager and executing the K3s agent binary directly, I identified an Invalid token format error. I extracted a pristine token from the master node using raw Linux text manipulation (cat piped into tr to strip hidden newline characters) to successfully authenticate the node.
• Systemd Configuration Drift: After a failed installation attempt, the systemctl agent continuously attempted to use a corrupted environment file. I resolved this by surgically overwriting the /etc/systemd/system/k3s-agent.service.env file using elevated command-line redirects and reloading the system daemon to force the correct credentials.
• Container Engine Disk Capacity Errors: After live-expanding a Proxmox VM disk, the containerd runtime crashed with an InvalidDiskCapacity error, reading the available space as 0. I orchestrated a clean node reboot and service restart to force the OS and the container runtime to re-poll the hardware specifications.

---

Active Workloads

This cluster serves as the highly available production backend for several personal development projects, including:

• Inventory Management API: Hosting the backend database and API endpoints for a custom NFC-triggered inventory tracking system.
• Automated Services: Containerized Discord bots deployed with zero-downtime rolling updates.

---

Repository Contents

• manifests/storage-test.yaml - PersistentVolumeClaim and Pod configuration for testing dynamic TrueNAS NFS provisioning.
• manifests/nginx-ingress.yaml - Traefik Ingress routing rules mapping a local domain to a backend web service.