🚀 NGINX Proxy PRD — Infrastructure as Code

Reverse‑proxy infrastructure built on Docker Compose and deployed via GitLab CI/CD.
High availability with Keepalived.

⚠️ Manual Configuration Required

Before deploying, you must:

Edit the files keepalived.conf and keepalived.conf to set your real notification emails and a strong VRRP password (auth_pass). Upload your SSL certificate and key files to the ssl/ directory. These files are required for secure HTTPS configuration in NGINX.

---

🗂 Repository Layout

.
├── conf.d/                 # NGINX vhosts (one .conf per site)
├── confs/
│   └── nginx.conf          # main nginx.conf
├── ssl/                    # certs/keys used by vhosts (mounted read‑only)
├── keepalived/
│   ├── master/keepalived.conf   # MASTER node config
│   └── backup/keepalived.conf   # BACKUP node config
    conf.d/                 # NGINX vhosts (one .conf per site)
    confs/
       nginx.conf          # main nginx.conf
    ssl/                    # Certificates/keys used by vhosts (mounted read-only)
    keepalived/
       master/keepalived.conf   # MASTER node configuration
       backup/keepalived.conf   # BACKUP node configuration
    nginx_cluster.yaml      # Docker Compose stack
    .gitlab-ci.yml          # CI/CD pipeline
├── nginx_cluster.yaml      # Docker Compose stack
└── .gitlab-ci.yml          # CI/CD pipeline

---

🧰 Runtime Environment

• Target hosts (Oracle Linux / systemd)
  • Docker Engine + Docker Compose Plugin
  • Keepalived
• Container image: nginx:latest
• Host directories
  • /opt/nginx-cluster (project files synced by CI)
• Container volumes (from nginx_cluster.yaml)
  • ./confs/nginx.conf:/etc/nginx/nginx.conf
  • ./conf.d:/etc/nginx/conf.d
  • ./ssl:/etc/nginx/ssl
  • /var/lib/nginx/logs:/var/log/nginx
  • /var/www/html/static:/var/www/html/static

---

🔐 Required CI Variables (GitLab → Settings » CI/CD » Variables)

• SSH_HOSTS – space‑separated IP list of the nodes (e.g., 192.168.0.111 192.168.0.112)
  Rule: the first IP is the MASTER (Keepalived) and the second IP is the BACKUP.
• SSH_USER – user with passwordless sudo (e.g., deployer)
• SSH_PRIVATE_KEY – OpenSSH private key not base64‑encoded (BEGIN/END)
• SSH_PORT – (optional) SSH port, default 22

🔒 Mark variables as Protected if you deploy only from a protected branch.

---

🧪→🚀 Pipeline (runs only on main)

Defined in .gitlab-ci.yml. Order of stages:

1. validate (nginx_config_test)
   Spins up a test nginx:latest, copies confs/, conf.d/, ssl/, then runs nginx -t.
   Any syntax error fails the pipeline.

2. check_or_install_docker (check_or_install_docker_on_vms)
   SSH into each host in SSH_HOSTS. Ensures/installs: docker-ce, docker-compose-plugin, rsync, nfs-utils, keepalived.
   Enables & starts docker.service.

3. sync_project_files
   Rsyncs the entire repo to /opt/nginx-cluster with safe perms (Du=rwx,Dgo=rx,Fu=rw,Fgo=r) and ownership ${SSH_USER}:${SSH_USER}.

4. deploy_keepalived
   Uploads and validates (keepalived -t) the proper file:

   • keepalived/master/keepalived.conf → 1st IP (MASTER)
   • keepalived/backup/keepalived.conf → 2nd IP (BACKUP)

   Replaces /etc/keepalived/keepalived.conf only if changed; then enable + reload (or restart) as needed.

5. deploy_systemd_service
   Installs/updates /etc/systemd/system/nginx-cluster.service:

   [Service]
   Type=oneshot
   WorkingDirectory=/opt/nginx-cluster
   ExecStart=/usr/bin/docker compose -f /opt/nginx-cluster/nginx_cluster.yaml up -d
   ExecStop=/usr/bin/docker compose -f /opt/nginx-cluster/nginx_cluster.yaml down
   RemainAfterExit=yes

   Performs daemon-reload, enable, and start/restart if required.

🧭 Branch policy: workflow at the top of .gitlab-ci.yml restricts the pipeline to branch main only.

---

🌐 How to Publish a New Site (vhost)

1. Create a file in conf.d/
   Example: conf.d/my-site.conf with a standard server { ... } block. For TLS, reference certs inside the container:

   ssl_certificate     /etc/nginx/ssl/my-site.crt;
   ssl_certificate_key /etc/nginx/ssl/my-site.key;

2. (Optional) Add certificates to ssl/
   Put the .crt and .key files there.
3. Local sanity check (optional)

   docker run --rm \
     -v $PWD/confs/nginx.conf:/etc/nginx/nginx.conf \
     -v $PWD/conf.d:/etc/nginx/conf.d \
     -v $PWD/ssl:/etc/nginx/ssl \
     nginx:latest nginx -t

4. Commit + push to main
   The validate stage blocks bad configs.
5. Deploy
   Repo is synced to /opt/nginx-cluster. The systemd unit runs the Compose stack.
   To apply vhost changes immediately without recreating the container:

   docker exec -it nginx-prd nginx -s reload
   # or
   sudo systemctl restart nginx-cluster

---

🛡 Keepalived Updates (VIP/priority, etc.)

1. Edit only:
   • keepalived/master/keepalived.conf (applied to 1st IP)
   • keepalived/backup/keepalived.conf (applied to 2nd IP)
2. Commit to main. The stage deploy_keepalived runs keepalived -t on the host, updates only if changed, and reloads/restarts the service.

⚠️ Tip: auth_pass in VRRP is limited to 8 chars by keepalived (older versions).

---

🧩 Compose Stack (nginx_cluster.yaml)

version: '3.1'
services:
  pwd-ws-hom:
    container_name: nginx-prd
    restart: always
    image: nginx:latest
    ports: ["80:80", "443:443"]
    volumes:
      - ./confs/nginx.conf:/etc/nginx/nginx.conf
      - ./conf.d:/etc/nginx/conf.d
      - ./ssl:/etc/nginx/ssl
      - /var/lib/nginx/logs:/var/log/nginx
      - /var/www/html/static:/var/www/html/static

ℹ️ Compose v2 treats version as obsolete; it is kept for compatibility.

---

🛠 Useful Ops (on the hosts)

# Compose service status
sudo systemctl status nginx-cluster

# Force update of the stack
sudo systemctl restart nginx-cluster

# Container logs
sudo docker logs -f nginx-prd

# Reload nginx config only
sudo docker exec -it nginx-prd nginx -s reload

# Keepalived status
sudo systemctl status keepalived

---

🚧 Common Pitfalls

• validate stage fails → NGINX syntax error in some .conf. Check job logs.
• Keepalived not starting → the job prints keepalived -t output; fix VRRP config (auth_pass, interface, VIPs).

---

🔒 Security Notes

• SSH_PRIVATE_KEY must be a raw OpenSSH key (BEGIN/END), without passphrase, and with correct line breaks (don’t base64 it).
• ${SSH_USER} needs passwordless sudo for the commands used by the pipeline.

---

🧭 End‑to‑End Flow

1. Commit to main → CI validates configs.
2. CI ensures Docker/Compose and Keepalived on nodes.
3. Repo is synced to /opt/nginx-cluster.
4. Keepalived (master/backup) is deployed and reloaded if changed.
5. nginx-cluster.service maintains the nginx:latest stack.
6. For vhost‑only changes, run nginx -s reload in the container (or restart the unit).

---

Made with ❤️ to keep your edge proxy HA, reproducible, and easy to operate.