Home Stack

Home Project Stack

The stack is deployed using Kubernetes cluster enabled using microk8s (https://microk8s.io/docs). microk8s is installed using snap package manger. Package is provided by Canonical (publisher of Ubuntu).
- Resources: quad-core ARMx64 processor with 8GB RAM
- Kernel: GNU/Linux 6.8.0-1015-raspi aarch64
- OS: Ubuntu 24.04.1

As of now it is deployed on 2 nodes cluster.

- Alok Singh

Table of contents

• Home Stack
  • Table of contents
  • Prerequisites
  • Deployment of home-stack Kubernetes Stack
    • Create Namespaces
    • Roll Binding for cluster admin user: alok
      • Setup remote user alok
    • Node Taint
      • Master Node
      • Worker Node
    • Kubernetes Dashboard
    • Kubernetes Metrics Server
    • Create ConfigMap
    • Create Secrets
    • Create Network policy
    • MySQL Service - Pod/Deployment/Service
    • Home Network Troubleshoot - Pod/Statefulset/Service
    • Home API Service - Pod/Deployment/Service
    • Home Email Service - Pod/Deployment/Service
    • Home Auth Service - Pod/Deployment/Service
    • Home Analytics Service - Pod/Deployment/Service
    • Home Search Service - Pod/Deployment/Service
    • Home Event Service - Pod/Deployment/Service
    • Home ETL Service - Pod/Statefulset/Service
    • Home GIT Commit CronJob (retired)
    • Dashboard Service - Pod/Deployment/Service
    • Jaeger Service
    • Mosquitto MqTT Service
    • IoT Telemetry Service
    • Delete Stack
  • Ingress
    • Ingress Create
    • Ingress Delete
    • Ingress Describe
    • Ingress Logs
  • Horizontal Autoscaling
    • Create Horizontal Pod Autoscaler
    • Manually Autoscale
    • Update Scale to 1
  • Miscellaneous commands
    • Client and Server version
    • API Resources
    • Get Node Details
    • Get Cluster Dump
    • Get all from all namespaces
    • Get all Services
    • Describe a Service
    • Get Pod Log
    • Describe a Pod
    • top a pod
    • Get All Pods under All Namespaces
    • Describe a spec
    • List all Docker images in Microk8s cluster (within the cluster node)
    • Prune Docker Images from Microk8s CLuster
      • Install crictl
      • Configure crictl for Microk8s
      • Prune Images
  • Service Mesh - Istio
    • Install
  • Backup
    • Config Map
    • Secrets
  • Network Monitoring
    • Kubeshark
      • Start Monitoring Pods
      • Stop Monitoring Pods
  • Deployment Architecture
    • Services

Prerequisites

• Kubernetes Setup on Raspberry Pi

---

Deployment of home-stack Kubernetes Stack

Create Namespaces

ssh alok@jgte "mkdir yaml"

scp yaml/namespace.yaml alok@jgte:yaml/

ssh alok@jgte "kubectl apply -f yaml/namespace.yaml"

Roll Binding for cluster admin user: alok

So that cluster operation can be performed by running kubectl remotely

scp yaml/home-user-rback-cluster-admin-user.yaml alok@jgte:yaml/

ssh alok@jgte "kubectl apply -f yaml/home-user-rback-cluster-admin-user.yaml"

Setup remote user alok

Please refer here

---

Node Taint

Master Node

at the end - to remove

kubectl taint nodes jgte nodeType=master:NoSchedule-

Worker Node

at the end - to remove

kubectl taint nodes khbr nodeType=worker:NoSchedule-

Kubernetes Dashboard

kubectl apply -f yaml/kubernetes-dashboard.yaml

Note: the dashboard service type is LoadBalancer and host IP (static) is assigned. The Dashboard can be access directly - https://jgte:8443/

kubectl delete -f yaml/kubernetes-dashboard.yaml

kubectl get all --namespace kubernetes-dashboard

kubectl get svc --namespace kubernetes-dashboard

kubectl apply -f yaml/kubernetes-dashboard-rback-dashboard-admin-user.yaml

kubectl create token k8s-dashboard-admin-user --duration=999999h -n kubernetes-dashboard

kubectl apply -f yaml/kubernetes-dashboard-rback-cluster-admin-user.yaml

kubectl create token k8s-dashboard-cluster-admin-user --duration=999999h -n kubernetes-dashboard

Notes:
- the last one doesnt have workloads get role
- use one of this token for Kubernetes Dashboard login

---

Kubernetes Metrics Server

kubectl apply -f yaml/metrix-server.yaml

kubectl delete -f yaml/metrix-server.yaml

kubectl get deployment metrics-server -n kube-system

kubectl top nodes

---

Create ConfigMap

kubectl apply -f yaml/config-map.yaml

Note: add/update below configs from backup ~/k8s

1. home-api-cofig (home-stack) 2. iot-secure-keystore-password 3. iot-secure-truststore-password
2. home-auth-cofig (home-stack) 5. application-security-jwt-secret 6. oauth-google-client-id 7. logging-level-com-alok
3. home-etl-cofig (home-stack) 9. git-bearer-token

Create Secrets

kubectl apply -f yaml/secrets.yaml

Create Network policy

kubectl apply -f yaml/networkpolicy.yaml

---

MySQL Service - Pod/Deployment/Service

ssh alok@jgte mkdir -p /home/alok/data/mysql

kubectl apply --validate=true --dry-run=client -f yaml/mysql-service.yaml 

kubectl apply -f yaml/mysql-service.yaml

kubectl delete -f yaml/mysql-service.yaml

kubectl exec -it pod/mysql-0 --namespace home-stack-db -- mysql -u root -p<<password>>

CREATE DATABASE `home-stack`;

kubectl exec -it pod/mysql-0 --namespace home-stack-db -- mysql -u root -p home-stack

kubectl logs pod/mysql-0 --namespace home-stack-db

mysql -u root -p home-stack --host 127.0.0.1 --port 32306

Note:

Run liquibase to create batch tables and add application users and roles

Follow the link to configure sqldeveloper on Mac to connect to MySQL server remotely

---

Home Network Troubleshoot - Pod/Statefulset/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-nw-tshoot.yaml 

kubectl apply -f yaml/home-nw-tshoot.yaml  --namespace=home-stack

kubectl delete -f yaml/home-nw-tshoot.yaml  --namespace=home-stack

kubectl exec -it pod/home-nw-tshoot-deployment-0 --namespace home-stack -- zsh

---

Home API Service - Pod/Deployment/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-api-service.yaml 

kubectl apply -f yaml/home-api-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-api-service.yaml  --namespace=home-stack

kubectl exec -it pod/home-api-deployment-0 --namespace home-stack -- bash

kubectl exec -it pod/home-api-deployment-0 --namespace home-stack -- tail -f /opt/logs/application.log

kubectl logs pod/home-api-deployment-0 --namespace home-stack

kubectl rollout restart statefulset.apps/home-api-deployment -n home-stack

---

Home Email Service - Pod/Deployment/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-email-service.yaml 

kubectl apply -f yaml/home-email-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-email-service.yaml  --namespace=home-stack

Home Auth Service - Pod/Deployment/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-auth-service.yaml 

kubectl apply -f yaml/home-auth-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-auth-service.yaml  --namespace=home-stack

kubectl exec -it pod/home-auth-deployment-0 --namespace home-stack -- bash

read instance

kubectl exec -it pod/home-auth-deployment-$instance --namespace home-stack -- tail -f /opt/logs/application.log

kubectl logs pod/home-auth-deployment-$instance --namespace home-stack

kubectl rollout restart statefulset.apps/home-api-deployment -n home-stack

---

Home Analytics Service - Pod/Deployment/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-analytics-service.yaml 

kubectl apply -f yaml/home-analytics-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-analytics-service.yaml  --namespace=home-stack

read instance

kubectl logs pod/home-analytics-deployment-$instance --namespace home-stack

kubectl exec -it pod/home-analytics-deployment-$instance --namespace home-stack -- bash

---

Home Search Service - Pod/Deployment/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-search-service.yaml 

kubectl apply -f yaml/home-search-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-search-service.yaml  --namespace=home-stack

read instance

kubectl logs pod/home-search-deployment-$instance --namespace home-stack

kubectl exec -it pod/home-search-deployment-$instance --namespace home-stack -- bash

Home Event Service - Pod/Deployment/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-event-service.yaml 

kubectl apply -f yaml/home-event-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-event-service.yaml  --namespace=home-stack

read instance

kubectl logs pod/home-event-deployment-$instance --namespace home-stack

kubectl exec -it pod/home-event-deployment-$instance --namespace home-stack -- bash

---

Home ETL Service - Pod/Statefulset/Service

kubectl apply --validate=true --dry-run=client -f yaml/home-etl-service.yaml 

kubectl apply -f yaml/home-etl-service.yaml  --namespace=home-stack

kubectl delete -f yaml/home-etl-service.yaml  --namespace=home-stack

kubectl exec -it pod/home-etl-deployment-0 --namespace home-stack -- bash

kubectl exec -it pod/home-etl-deployment-0 --namespace home-stack -- tail -f /opt/logs/application.log

kubectl logs pod/home-etl-deployment-0 --namespace home-stack

kubectl rollout restart statefulset.apps/home-api-deployment -n home-stack

---

Home GIT Commit CronJob (retired)

kubectl apply --validate=true --dry-run=client -f yaml/git-commit-cronjob.yaml 

kubectl apply -f yaml/git-commit-cronjob.yaml  --namespace=home-stack

kubectl delete -f yaml/git-commit-cronjob.yaml  --namespace=home-stack

---

Dashboard Service - Pod/Deployment/Service

kubectl apply -f yaml/dashboard-nginx-config-map.yaml

kubectl apply --validate=true --dry-run=client -f yaml/dashboard-service.yaml 

kubectl apply -f yaml/dashboard-service.yaml

kubectl delete -f yaml/dashboard-service.yaml

kubectl exec -it deployment.apps/dashboard-deployment --namespace home-stack-dmz -- /bin/sh

kubectl logs deployment.apps/dashboard-deployment --namespace home-stack-dmz

---

Jaeger Service

kubectl apply --validate=true --dry-run=client -f yaml/jaeger-all-in-one-template.yml 

kubectl apply -f yaml/jaeger-all-in-one-template.yml  --namespace=home-stack

kubectl delete -f yaml/jaeger-all-in-one-template.yml  --namespace=home-stack

---

Mosquitto MqTT Service

kubectl apply -f yaml/iot-config-map.yaml

kubectl apply --validate=true --dry-run=client -f yaml/mosquitto-service.yaml 

kubectl create secret tls mosquitto-secret --cert=../iot-home-stack/secret/server.crt --key=../iot-home-stack/secret/server.key --namespace=home-stack-iot

kubectl create secret generic mosquitto-ca-secret --from-file=../iot-home-stack/secret/mqtt-signer-ca.crt --namespace=home-stack-iot

kubectl delete secret mosquitto-acl-secret --namespace=home-stack-iot

kubectl create secret generic mosquitto-acl-secret --from-file=../iot-home-stack/secret/acl.conf --namespace=home-stack-iot

kubectl apply -f yaml/iot-mosquitto-service.yaml  --namespace=home-stack-iot

kubectl delete -f yaml/iot-mosquitto-service.yaml --namespace=home-stack-iot

IoT Telemetry Service

kubectl apply -f yaml/iot-telemetry-config-map.yaml

kubectl create secret generic iot-telemetry-secret --from-file=keystore.jks=../iot-home-stack/secret/mqtt.client.home-telemetry-svc.jks --namespace=home-stack-iot

kubectl apply --validate=true --dry-run=client -f yaml/iot-telemetry-service.yaml 

kubectl apply -f yaml/iot-telemetry-service.yaml  --namespace=home-stack-iot

kubectl delete -f yaml/iot-telemetry-service.yaml --namespace=home-stack-iot

---

Delete Stack

kubectl delete namespace home-stack-dmz

kubectl delete namespace home-stack 

kubectl delete namespace home-stack-db 

---

Ingress

Ingress Create

kubectl apply -f yaml/ingress.yaml

Ingress Delete

kubectl delete -f yaml/ingress.yaml

Ingress Describe

kubectl get ingress -n home-stack-dmz

kubectl describe ingress -n home-stack-dmz

kubectl describe ingress ingress-home-jgte --namespace home-stack-dmz

kubectl get all --namespace ingress

kubectl describe daemonset.apps/nginx-ingress-microk8s-controller --namespace ingress

kubectl describe pod/nginx-ingress-microk8s-controller-8wmwc --namespace ingress 

Ingress Logs

kubectl get all --namespace ingress

kubectl logs nginx-ingress-microk8s-controller-8wmwc  --namespace ingress

---

Horizontal Autoscaling

Create Horizontal Pod Autoscaler

kubectl apply --validate=true --dry-run=client -f yaml/home-hpa.yaml

kubectl apply -f yaml/home-hpa.yaml  --namespace=home-stack

kubectl get hpa

kubectl describe hpa home-auth-hpa

kubectl describe hpa home-api-hpa

kubectl describe hpa home-analytics-hpa

Manually Autoscale

kubectl autoscale deployment dashboard-deployment --min=2 --max=3 -n home-stack

kubectl get hpa --namespace home-stack

Update Scale to 1

kubectl edit hpa dashboard-deployment --namespace home-stack

kubectl scale -n home-stack deployment dashboard-deployment --replicas=1

---

Miscellaneous commands

Client and Server version

kubectl version --output=json

API Resources

kubectl api-resources

Get Node Details

This gives details about nodes including images in local

kubectl get nodes -o yaml

kubectl describe nodes

kubectl get ResourceQuota

Get Cluster Dump

This gives cluster dump including all pods log

kubectl cluster-info dump > ~/k8s/cluster-dump.log

Get all from all namespaces

kubectl get all --all-namespaces

Get all Services

kubectl get svc --all-namespaces 

Describe a Service

kubectl describe svc dashboard-service --namespace home-stack-dmz

kubectl describe svc kubernetes-dashboard --namespace kubernetes-dashboard

Get Pod Log

kubectl logs pod/dashboard-deployment-65cf5b8858-7x8z8 --namespace home-stack

Describe a Pod

kubectl describe pod home-etl-deployment-0  --namespace=home-stack

top a pod

kubectl top pods

kubectl top pod home-etl-deployment-0 --containers

Get All Pods under All Namespaces

kubectl get po -A -o wide

Describe a spec

kubectl api-resources 

kubectl explain --api-version="networking.k8s.io/v1" NetworkPolicy.spec

kubectl explain --api-version="networking.k8s.io/v1" NetworkPolicy.spec.ingress

kubectl explain --api-version="batch/v1beta1" cronjobs.spec

kubectl get crd 

kubectl explain --api-version="apiregistration.k8s.io/v1" APIService

kubectl explain --api-version="apiextensions.k8s.io/v1" CustomResourceDefinition

List all Docker images in Microk8s cluster (within the cluster node)

sudo microk8s ctr images ls

Prune Docker Images from Microk8s CLuster

Install crictl

VERSION="v1.26.0" # check latest version in /releases page
curl -L https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-${VERSION}-linux-arm64.tar.gz --output crictl-${VERSION}-linux-arm64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-arm64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-arm64.tar.gz

Configure crictl for Microk8s

sudo vim /etc/crictl.yaml 

runtime-endpoint: unix:///var/snap/microk8s/common/run/containerd.sock
image-endpoint: unix:///var/snap/microk8s/common/run/containerd.sock
timeout: 10
debug: true

Prune Images

sudo crictl rmi --prune

kubectl cheat sheet - https://kubernetes.io/docs/reference/kubectl/cheatsheet/

---

Service Mesh - Istio

Install

---

To be explored - seems microk8s isteo addon not supported for ARMx64 architecture. Where the same is supported for minikube.

---

Backup

Config Map

This is needed as some config items are directly updated in the cluster through Kubernetes Dashboard for security reason

kubectl get configmap --namespace=home-stack stmt-parser-cofig -o yaml > ~/k8s/stmt-parser-cofig.yaml
kubectl get configmap --namespace=home-stack home-etl-cofig -o yaml > ~/k8s/home-etl-cofig.yaml
kubectl get configmap --namespace=home-stack home-api-cofig -o yaml > ~/k8s/home-api-cofig.yaml
kubectl get configmap --namespace=home-stack home-auth-cofig -o yaml > ~/k8s/home-auth-cofig.yaml
kubectl get configmap --namespace=home-stack dashboard-cofig -o yaml > ~/k8s/dashboard-cofig.yaml
kubectl get configmap --namespace=home-stack home-common-cofig -o yaml > ~/k8s/home-common-cofig.yaml
kubectl get configmap --namespace=home-stack-dmz nginx-conf -o yaml > ~/k8s/nginx-conf.yaml
kubectl get configmap --namespace=home-stack home-email-cofig  -o yaml > ~/k8s/home-email-cofig.yaml

Secrets

This is needed as some secret items are directly updated in the cluster through Kubernetes Dashboard for security reason

kubectl get secrets --namespace=home-stack mysql-secrets -o yaml > ~/k8s/mysql-secrets.yaml
kubectl get secrets --namespace=home-stack-db mysql-secrets -o yaml > ~/k8s/mysql-secrets-db.yaml

---

Network Monitoring

Kubeshark

Start Monitoring Pods

kubeshark tap

Kubeshark dashboard is accessible using http://localhost:8899

Stop Monitoring Pods

kubeshark clean

---

Deployment Architecture

Services

Application	Description	Service Type	Deployment/StatefulSet/CronJob/DaemonSet	URL	Comments
Home ETL Service	ETL for bank statement and other sources	ClusterIP (Headless)	StatefulSet	/home/etl	NA
Home Auth Service	Home AuthN and AuthZ service	ClusterIP	Deployment	/home/api	GraalVM based native Image
Home API Service	API for Bank/Expense/Tax/Investment/etc...	ClusterIP	Deployment	/home/api	GraalVM based native Image
Home Analytics Service	gRPC interface to categorize expense	ClusterIP	Deployment	/home/api	GraalVM based native Image
Home Email Service	IMAP to read bank transactions and SMTP to send mail	ClusterIP	Deployment	/home/api	GraalVM based native Image
Home Dashboard	ReactJS App on Nginx	NodePort	Deployment	http://jgte:30080 or https://jgte	- For multinode deployment Interface has to be changed to ClusterIP and put behind Ingress - externalTrafficPolicy: Local to disable SNATing
Home GIT Cronjob	Cronjob to update GIT with uploaded statement (not in use)	None	CronJob	NA	NA
Database	MySQL	NodePort	StatefulSet	jdbc:mysql://mysql:3306/home-stack	- NodePort because I want to access SQL from outside of the cluster
Kubernetes Dashboard		LoadBalancer (static IP)	Deployment	https://jgte:8443/
Kubernetes Matrix	Generating resource utilization matrix	ClusterIP	Deployment	NA
Kubernetes Matrix Scraper	Matrix scrapper from pods	ClusterIP	Deployment	NA
Jaeger Dashboard		NodePort	Deployment	http://jgte:31686/
Ingress Controller	Nginx Ingress Controller	NodePort	DaemonSet	Port: 443	API/ETL/Dashboard are behind Nginx but still we have Dashboard accessible directly (from mobile cant access host name - require local DNS server)

---

graph LR
    A[Write Code] --> B{Does it work?}
    B -- Yes --> C[Great!]
    B -- No --> D[Google]
    D --> A

Loading