Simple Local Cluster

Easy-ish to use containerized cluster setup using Docker Compose and k3s.

Table of Contents

1. Motivation
2. Usage
   1. Configuration
   2. Host Setup
   3. Running

Motivation

To develop apps to be deployed to Kubernetes one might need a simple way to create a cluster with certain capabilities.

To do this there is many great options like Kind, K3D and many more.

So why another option?

This repository offers a simple Docker Compose based solution with these features:

• Direct access to app via port mapping, no additional Load Balancer needed
• Basic PKI infrastructure for TLS/HTTPS
• Simple DNS Server

Usage

Prerequisites

You will need:

• Docker
• Docker Compose
• Make
• kubectl

Configuration

Copy example.env to .env and change to fit your needs.

These are the defaults:

• PRIVATE_IP=127.0.0.1

  The kubeapi is bound to this with port 6443

• PUBLIC_IP=192.168.0.0

  This is the "public" facing IP to point your browser to when accessing App UIs running in the cluster, your LAN/WLAN IP. Note: this can't be 127.0.0.1.

• DOMAIN=my-project.intern

  The domain name where our Apps should be reachable, the DNS container will resolve everything under this domain to PUBLIC_IP.

Host Setup

To make use of the included DNS server your host system needs to know about it, so it knows that every DNS request for DOMAIN(from your .env) should be forwarded to PUBLIC_IP:1053.

If you are using Linux you might have one of the following options available.

Attention: these examples are held very simple and might absolutely not fit your Linux distro. Make sure to consult its documentation if unsure.

dnsmasq

Create the file /etc/dnsmasq.d/my-project.intern(or what you set DOMAIN to), with the following line(fix to match your DOMAIN and PRIVATE_IP):

server=/my-project.intern/127.0.0.1:1053

Then restart dnsmasq with

systemctl restart dnsmasq.service

See it's manpage for details. and/or consult your distro's documentation.

systemd-resolved

This will use sudo to write a file unter /etc/systemd/resolved.conf.d/ and restart systemd-resolved via systemctl.

As this writes into /etc with root permissions, make sure you check the Makefile target.

make write-resolved-conf.d

This This is (not yet) working, see systemd-resolved MR.

Edit the file /etc/systemd/resolved.conf, find the [Resolve] block and add this (fix to match your DOMAIN and PUBLIC_IP):

DNS=127.0.0.1:1053#my-project.intern

Running

Start cluster with:

make up

Check logs:

make tail

Get kubeconfig with:

make get-kubeconfig

Install 3rd party manifests(only need on first startup):

make install-3rdparty

Get pods:

kubectl --kubeconfig .kubeconfig get pods -A

PKI

If you point your browser at http://$DOMAIN(your domain name from .env, without HTTPS(!)), you will find a link to download the certificate of a CA used in the cluster for ingresses. This can be installed in the browser or some local certificate store.

Cleanup

Run make rm to delete everything but the k3s server volume.

Troubleshooting

If your kubeapi client runs into something like this:

Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority

You might have a leftover .kubeconfig file, make sure to delete it manually and run make get-kubeconfig again.