VPS Deployment Guide v2

Step-by-step untuk reset dan deploy ulang K3s cluster dari awal.

Prerequisites

Di VPS (staging/production):

Ubuntu 24.04 LTS
SSH access dengan root/sudo
Disk partitions sudah ready:
- Staging: /dev/sdb1 → /staging-goapps-backup
- Production: /dev/sdb1 → /goapps-backup
SSL certificates: ssl-bundle.crt dan ssl-bundle.key

Di local machine:

Git repo goapps-infra sudah di-push ke GitHub

Step 1: Reset K3s (Jika Ada Instalasi Lama)

# SSH ke VPS
ssh deploy@[staging-]goapps.mutugading.com

# Cek apakah K3s sudah terinstall
which k3s

# Jika ada, jalankan uninstall script
sudo /usr/local/bin/k3s-uninstall.sh

# Verifikasi sudah bersih
ls /var/lib/rancher  # Harusnya: No such file or directory
ls /etc/rancher       # Harusnya: No such file or directory

Step 2: Clone Repository

# Clone goapps-infra
cd ~
git clone https://github.com/mutugading/goapps-infra.git
cd goapps-infra

# Pastikan scripts executable
chmod +x scripts/*.sh

Step 3: Bootstrap K3s Cluster

# Jalankan bootstrap (install K3s, Helm, namespaces)
./scripts/bootstrap.sh

# Tunggu sampai selesai, verifikasi:
kubectl get nodes
kubectl get namespaces

Step 4: Create Secrets (MANUAL - Tidak di Git!)

Lihat template di base/secrets/secrets-template.yaml untuk referensi.

PostgreSQL

kubectl create secret generic postgres-secret -n database \
  --from-literal=POSTGRES_USER=goapps_admin \
  --from-literal=POSTGRES_PASSWORD='<STRONG_PASSWORD>' \
  --from-literal=POSTGRES_DB=goapps

MinIO

kubectl create secret generic minio-secret -n minio \
  --from-literal=MINIO_ROOT_USER=admin \
  --from-literal=MINIO_ROOT_PASSWORD='<STRONG_PASSWORD>'

# Copy ke namespace database
kubectl get secret minio-secret -n minio -o yaml | \
  sed 's/namespace: minio/namespace: database/' | \
  kubectl apply -f -

RabbitMQ

kubectl create secret generic rabbitmq-secret -n database \
  --from-literal=RABBITMQ_USER=goapps \
  --from-literal=RABBITMQ_PASSWORD='<STRONG_PASSWORD>'

Oracle Credentials (Multiple Schemas)

kubectl create secret generic oracle-credentials -n go-apps \
  --from-literal=ORACLE_HOST='<ORACLE_IP>' \
  --from-literal=ORACLE_PORT='1521' \
  --from-literal=ORACLE_SERVICE='ORCLPDB1' \
  --from-literal=ORACLE_MGTHRIS_USER='mgthris' \
  --from-literal=ORACLE_MGTHRIS_PASSWORD='<PASSWORD>' \
  --from-literal=ORACLE_MGTAPPS_USER='mgtapps' \
  --from-literal=ORACLE_MGTAPPS_PASSWORD='<PASSWORD>' \
  --from-literal=ORACLE_MGTDAT_USER='mgtdat' \
  --from-literal=ORACLE_MGTDAT_PASSWORD='<PASSWORD>'

TLS Certificate

# Create TLS secret untuk Ingress
kubectl create secret tls goapps-tls -n ingress-nginx \
  --cert=ssl-bundle.crt \
  --key=ssl-bundle.key

# Copy ke namespace monitoring
kubectl get secret goapps-tls -n ingress-nginx -o yaml | \
  sed 's/namespace: ingress-nginx/namespace: monitoring/' | \
  kubectl apply -f -

# Create TLS secret untuk Kubernetes Dashboard
kubectl create secret tls dashboard-tls -n kubernetes-dashboard \
  --cert=ssl-bundle.crt \
  --key=ssl-bundle.key

Grafana Admin & SMTP

kubectl create secret generic grafana-admin-secret -n monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password='<GRAFANA_PASSWORD>'

kubectl create secret generic grafana-smtp-secret -n monitoring \
  --from-literal=password='<SMTP_PASSWORD>'

Backblaze B2 (Cloud Backup)

kubectl create secret generic s3-cloud-credentials -n database \
  --from-literal=S3_ENDPOINT='s3.us-west-004.backblazeb2.com' \
  --from-literal=S3_BUCKET='goapps-backups' \
  --from-literal=AWS_ACCESS_KEY_ID='<B2_KEY_ID>' \
  --from-literal=AWS_SECRET_ACCESS_KEY='<B2_APP_KEY>'

Container Registry Pull Secret

kubectl create secret docker-registry ghcr-secret -n go-apps \
  --docker-server=ghcr.io \
  --docker-username='<GITHUB_USERNAME>' \
  --docker-password='<GITHUB_TOKEN>'

Step 5: Install Monitoring Stack

./scripts/install-monitoring.sh

# Tunggu semua pods ready
kubectl get pods -n monitoring -w

# Akses Grafana (via Ingress atau port-forward)
kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80

Step 6: Install ArgoCD (GitOps)

./scripts/install-argocd.sh

# Catat password dari output!

# Akses ArgoCD UI:
kubectl port-forward svc/argocd-server -n argocd 8080:443

Step 7: Apply Base Infrastructure

# Apply semua base configs
make apply-base

# Atau manual:
kubectl apply -k base/namespaces/
kubectl apply -k base/database/  # includes PostgreSQL, PgBouncer, Redis, RabbitMQ
kubectl apply -k base/backup/
kubectl apply -k base/observability/  # Jaeger

# Apply alert rules
kubectl apply -f base/monitoring/alert-rules/

Verifikasi Services

# Database Layer
kubectl get pods -n database

# Jaeger Tracing
kubectl get pods -n observability

# Check Redis & RabbitMQ
kubectl exec -it deploy/redis -n database -- redis-cli ping  # PONG

Step 8: Apply ArgoCD Applications

kubectl apply -f argocd/apps/

# Cek status
kubectl get applications -n argocd

Step 9: Deploy Services (Via ArgoCD)

ArgoCD akan auto-sync dari Git berdasarkan branch/tag:

Staging: main branch → auto deploy
Production: Release tags (v1.0.0) → manual approval

Step 10: Run Database Migrations & Seeds

Tables are NOT auto-created on deploy. You must run migrations and seeds manually via K8s Jobs after each service is deployed for the first time (or when new migrations are added).

How It Works

Each backend service has:

migrate-job.yaml — K8s Job that runs golang-migrate to apply SQL migrations
seed-job.yaml — K8s Job that runs the Go seeder binary to insert initial data
Setup script (scripts/<service>-setup.sh) — convenience wrapper that runs both

Seeds use ON CONFLICT DO NOTHING, so they are idempotent (safe to re-run).

Finance Service

# From goapps-infra directory on the VPS
./scripts/finance-setup.sh goapps-staging           # migrate + seed
./scripts/finance-setup.sh goapps-production         # migrate + seed
./scripts/finance-setup.sh goapps-staging migrate    # migrate only
./scripts/finance-setup.sh goapps-staging seed       # seed only

What gets seeded: UOM master data, RM Category master data.

IAM Service

./scripts/iam-setup.sh goapps-staging               # migrate + seed
./scripts/iam-setup.sh goapps-production             # migrate + seed
./scripts/iam-setup.sh goapps-staging migrate        # migrate only
./scripts/iam-setup.sh goapps-staging seed           # seed only

What gets seeded: Admin user, roles, permissions, menus (3-level hierarchy), CMS content (pages, sections, settings).

When to Run

Scenario	Action
First deploy of a service	`./scripts/<service>-setup.sh <namespace>` (migrate + seed)
New migration added (new tables/columns)	`./scripts/<service>-setup.sh <namespace> migrate`
New seed data added (e.g., CMS content)	`./scripts/<service>-setup.sh <namespace> seed`
Routine deploy (no schema changes)	Nothing — ArgoCD handles the deployment, no migration needed

Prerequisites

The service deployment must already be running (the script reads the image tag from the running deployment)
For custom admin credentials in production, create iam-seed-secret first (see seed-job.yaml)

Troubleshooting

# Check migration job status
kubectl get jobs -n goapps-staging | grep migrate

# Check seed job logs
kubectl logs job/iam-seed -n goapps-staging

# Check migration pod logs (shows actual error)
kubectl logs -n goapps-staging -l job-name=iam-migrate

# Delete old failed jobs before re-running
kubectl delete job iam-migrate iam-seed -n goapps-staging --ignore-not-found

Dirty Migration Fix

If migration fails with error: Dirty database version N. Fix and force version.:

# 1. Check current state
kubectl exec -it postgres-0 -n database -- psql -U postgres -d goapps -c \
  "SELECT version, dirty FROM schema_migrations_iam;"

# 2. Determine the correct version:
#    - If tables already exist in DB, set version to the LAST SUCCESSFULLY APPLIED migration
#    - Check which tables exist to determine the right version number
#    - Example: if all 13 IAM migrations were applied, set version=13

# 3. Fix the dirty flag AND set correct version
kubectl exec -it postgres-0 -n database -- psql -U postgres -d goapps -c \
  "UPDATE schema_migrations_iam SET version = <correct_version>, dirty = false;"

# 4. Re-run migration
kubectl delete job iam-migrate -n goapps-staging --ignore-not-found
./scripts/iam-setup.sh goapps-staging migrate

Common cause: Migration Job crashed/timed out mid-execution, or tables were created outside of golang-migrate (manual SQL, seed job). The schema_migrations_* table tracks which version was last applied — if it doesn't match reality, you must manually correct it.

For finance service, replace schema_migrations_iam with schema_migrations_finance.

Migration Files Reference

Service	Migration	Description
Finance	000001	UOM tables + indexes
Finance	000002	Audit logs
Finance	000003	RM Category tables
IAM	000001-000011	Auth, users, roles, permissions, organizations, menus, sessions, audit
IAM	000012	CMS tables (mst_cms_page, mst_cms_section, mst_cms_setting)
IAM	000013	CMS menu entries + permissions
Finance	000004	Parameter tables (mst_parameter) + indexes
IAM	000014	Parameter menu entry + 6 permissions + SUPER_ADMIN role_permissions

Step 11: Verifikasi Final

Checklist

Environment-Specific Paths

Environment	Backup Mount	Clone Command
Staging	`/staging-goapps-backup`	Use `overlays/staging/backup-patch.yaml`
Production	`/goapps-backup`	Use `overlays/production/backup-patch.yaml`

Troubleshooting

Pod tidak start

kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>

Database connection issues

kubectl run test-pg --rm -it --image=postgres:18-alpine -- \
  psql -h pgbouncer.database -U goapps_admin -d goapps

Oracle connectivity test

# Check outbound to Oracle port
kubectl run test-net --rm -it --image=busybox -- /bin/sh -c \
  "nc -vz <ORACLE_IP> 1521"

Quick Commands Reference

Command	Description
`make status`	Cluster overview
`make logs-postgres`	PostgreSQL logs
`make backup-now`	Manual backup trigger
`make port-forward-grafana`	Access Grafana locally
`make port-forward-argocd`	Access ArgoCD locally
`make port-forward-jaeger`	Access Jaeger UI

Architecture Summary

┌─────────────────────────────────────────────────────────────────┐
│                        VPS Cluster                              │
├─────────────┬───────────────────────────────────────────────────┤
│ Namespace   │ Components                                        │
├─────────────┼───────────────────────────────────────────────────┤
│ database    │ PostgreSQL, PgBouncer, Redis, RabbitMQ, Exporter │
│ minio       │ MinIO (Object Storage)                           │
│ monitoring  │ Prometheus, Grafana, Loki, Promtail, Alertmanager│
│ observability│ Jaeger (Tracing)                                 │
│ argocd      │ ArgoCD (GitOps)                                  │
│ go-apps     │ finance-service, iam-service, etc.               │
│ ingress-nginx│ NGINX Ingress Controller                        │
│ k8s-dashboard│ Kubernetes Dashboard                            │
└─────────────┴───────────────────────────────────────────────────┘

FilesExpand file tree

deployment-guide.md

Latest commit

History

deployment-guide.md

File metadata and controls

VPS Deployment Guide v2

Prerequisites

Step 1: Reset K3s (Jika Ada Instalasi Lama)

Step 2: Clone Repository

Step 3: Bootstrap K3s Cluster

Step 4: Create Secrets (MANUAL - Tidak di Git!)

PostgreSQL

MinIO

RabbitMQ

Oracle Credentials (Multiple Schemas)

TLS Certificate

Grafana Admin & SMTP

Backblaze B2 (Cloud Backup)

Container Registry Pull Secret

Step 5: Install Monitoring Stack

Step 6: Install ArgoCD (GitOps)

Step 7: Apply Base Infrastructure

Verifikasi Services

Step 8: Apply ArgoCD Applications

Step 9: Deploy Services (Via ArgoCD)

Step 10: Run Database Migrations & Seeds

How It Works

Finance Service

IAM Service

When to Run

Prerequisites

Troubleshooting

Dirty Migration Fix

Migration Files Reference

Step 11: Verifikasi Final

Checklist

Environment-Specific Paths

Troubleshooting

Pod tidak start

Database connection issues

Oracle connectivity test

Quick Commands Reference

Architecture Summary