refactor: modernize Z3 stack with upstream repos and Zebra health checks (#4)

* refactor: modernize Z3 stack with upstream repos and Zebra health checks

Update the Z3 stack to use official upstream repositories and their
corresponding Dockerfiles instead of building from personal forks with
specific branches. This refactor also implements proper Zebra configuration
using its new environment variable system and health check endpoints.

Major changes:

1. Use official upstream repositories with default branches
   - Zebra: ZcashFoundation/zebra (main branch)
   - Zaino: zingolabs/zaino (dev branch)
   - Zallet: zcash/wallet (main branch)
   - Updated Dockerfile paths to match each project's structure

2. Implement Zebra's environment variable configuration system
   - Use config-rs format: ZEBRA_SECTION__KEY for Zebra configuration
   - Implement Z3_* prefix for Docker Compose infrastructure variables
   - Three-tier variable hierarchy to prevent config collisions:
     * Z3_* for infrastructure (volumes, ports, Docker Compose only)
     * Shared variables (NETWORK_NAME, etc.) remapped per service
     * Service-specific (ZEBRA_*, ZAINO_*, ZALLET_*) for app config

3. Leverage Zebra's new health check endpoints
   - Use /ready endpoint for production (synced near network tip)
   - Use /healthy endpoint for development (has peer connections)
   - Implement two-phase deployment pattern for blockchain sync
   - Configure healthcheck with 90s start_period for cached state

4. Improve security and documentation
   - Add fix-permissions.sh for proper UID/GID ownership setup
   - Document specific UIDs/GIDs for each service
   - Add warnings about unstable development branches
   - Update deployment documentation for fresh sync scenarios

5. Remove hardcoded paths and configurations
   - Eliminate user-specific paths from docker-compose.yml
   - Use environment variables for all customizable settings
   - Add docker-compose.override.yml.example for development mode

* fix(compose): workaround through zallet limitations

* fix(zallet): Use Zebra as the indexer, and update docs

* chore: reduce logging

* docs: update README.md

---------

Co-authored-by: Gustavo Valverde <gustavovalverde@unknowncad5ab0a9f79.lan>

Author: gustavovalverde

.env

@@ -2,27 +2,104 @@
 
 # z3/docker-compose.yml Environment Variables
 
-# --- Common Configuration ---
-# Network name for all services (e.g., Mainnet, Testnet, Regtest). Referenced by Zebra, Zaino, and Zallet.
-NETWORK_NAME=Testnet
-# Globally enables RPC cookie authentication. If true, Zebra generates a .cookie file. 
-# Zaino's use of this cookie (via ZAINO_VALIDATOR_COOKIE_AUTH_ENABLE in compose) is conditional on its ZAINO_VALIDATOR_LISTEN_ADDRESS.
+# =============================================================================
+# VARIABLE HIERARCHY QUICK REFERENCE
+# =============================================================================
+# For detailed explanation, see README.md "Understanding the Variable Hierarchy"
+#
+# Three-tier system to avoid collisions:
+#   Z3_*        - Infrastructure (volumes, ports) - Docker Compose only
+#   (no prefix) - Shared config (remapped per service)
+#   ZEBRA_*     - Zebra app config (ZEBRA_SECTION__KEY format)
+#   ZAINO_*     - Zaino app config
+#   ZALLET_*    - Zallet app config
+#
+# =============================================================================
+# Z3 Stack Infrastructure Configuration
+# =============================================================================
+# These Z3_* variables control Docker Compose volume mounts and are NEVER
+# passed to containers. They avoid collision with service configs (ZEBRA_*,
+# ZAINO_*, ZALLET_*).
+#
+# Default: Docker named volumes (recommended - managed by Docker, no permission issues)
+# Advanced: Local directories (see README.md "Advanced: Local Directories" section)
+#
+# To use local directories:
+#   1. Choose appropriate paths for your OS (see README for suggestions)
+#   2. Create directories: mkdir -p /your/chosen/path
+#   3. Fix permissions: ./fix-permissions.sh <service> /your/chosen/path
+#   4. Update variables below with your paths
+#
+# Security Requirements:
+#   - Zebra:  UID=10001, GID=10001, permissions=700
+#   - Zaino:  UID=1000,  GID=1000,  permissions=700
+#   - Zallet: UID=65532, GID=65532, permissions=700
+#   - Cookie: Keep as Docker volume (recommended) to avoid cross-user issues
+#
+# WARNING: Never use 755 or 777 permissions - they expose your data!
+
+# Zebra blockchain state directory
+# Default: zebra_data (Docker named volume)
+Z3_ZEBRA_DATA_PATH=zebra_data
+
+# Shared cookie authentication directory (used by Zebra and Zaino)
+# Default: shared_cookie_volume (Docker named volume)
+# NOTE: For local directories, Zebra (10001) writes and Zaino (1000) reads.
+#       Consider using Docker volume (default) to avoid cross-user permission issues.
+#       If using local dir, you'll need to set up ACLs or a shared group.
+Z3_COOKIE_PATH=shared_cookie_volume
+
+# Zaino indexer data directory
+# Default: zaino_data (Docker named volume)
+Z3_ZAINO_DATA_PATH=zaino_data
+
+# Zallet wallet data directory
+# Default: zallet_data (Docker named volume)
+Z3_ZALLET_DATA_PATH=zallet_data
+
+# =============================================================================
+# Common Configuration
+# =============================================================================
+# Shared variables used by multiple services, mapped in docker-compose.yml:
+#   NETWORK_NAME → ZEBRA_NETWORK__NETWORK, ZAINO_NETWORK, ZALLET_NETWORK
+#   ENABLE_COOKIE_AUTH → ZEBRA_RPC__ENABLE_COOKIE_AUTH, ZAINO_VALIDATOR_COOKIE_AUTH
+#   COOKIE_AUTH_FILE_DIR → ZEBRA_RPC__COOKIE_DIR, ZAINO_VALIDATOR_COOKIE_PATH
+
+# Network name for all services (e.g., Mainnet, Testnet, Regtest)
+NETWORK_NAME=Mainnet
+# Globally enables RPC cookie authentication
 ENABLE_COOKIE_AUTH=true
-# In-container directory for the .cookie authentication file (e.g., /var/run/auth). 
-# Zebra writes its cookie here; Zaino reads from this location via its ZAINO_VALIDATOR_COOKIE_PATH in compose.
+# In-container directory for the .cookie authentication file
 COOKIE_AUTH_FILE_DIR=/var/run/auth
 
-# --- Zebra Configuration ---
-# Zebra Rust log level
-ZEBRA_RUST_LOG=info
-# Zebra's internal RPC port. Zaino connects to this port on the 'zebra' service hostname.
-ZEBRA_RPC_PORT=18232
-# Zebra host RPC port (for external access to Zebra)
-ZEBRA_HOST_RPC_PORT=18232
+# =============================================================================
+# Zebra Configuration
+# =============================================================================
+# Zebra logging (will be mapped to RUST_LOG in container)
+Z3_ZEBRA_RUST_LOG=info
+# Zebra tracing filter (config-rs format: ZEBRA_TRACING__FILTER)
+ZEBRA_TRACING__FILTER=info
+# Zebra RPC listen address (config-rs format: ZEBRA_RPC__LISTEN_ADDR)
+ZEBRA_RPC__LISTEN_ADDR=0.0.0.0:18232
+# Zebra state cache directory (config-rs format: ZEBRA_STATE__CACHE_DIR)
+ZEBRA_STATE__CACHE_DIR=/home/zebra/.cache/zebra
+# Zebra health endpoint configuration
+ZEBRA_HEALTH__LISTEN_ADDR=0.0.0.0:8080
+ZEBRA_HEALTH__MIN_CONNECTED_PEERS=1
+ZEBRA_HEALTH__READY_MAX_BLOCKS_BEHIND=2
+ZEBRA_HEALTH__ENFORCE_ON_TEST_NETWORKS=false
+# Infrastructure: Zebra RPC port (used in Docker Compose port mappings and service discovery)
+Z3_ZEBRA_RPC_PORT=18232
+# Infrastructure: Zebra host RPC port (for external access to Zebra)
+Z3_ZEBRA_HOST_RPC_PORT=18232
+# Infrastructure: Zebra host health port (for external access to health endpoints)
+Z3_ZEBRA_HOST_HEALTH_PORT=8080
 
-# --- Zaino Configuration ---
+# =============================================================================
+# Zaino Configuration
+# =============================================================================
 # Zaino Rust log level
-ZAINO_RUST_LOG=trace,hyper=info
+ZAINO_RUST_LOG=info,reqwest=warn,hyper_util=warn
 # Zaino's internal gRPC port. Zallet connects to this port on the 'zaino' service hostname.
 ZAINO_GRPC_PORT=8137
 # Enable/disable Zaino's JSON-RPC service
@@ -53,9 +130,11 @@ ZAINO_CONF_PATH=/home/zaino/.config/zaino/zindexer.toml
 # Zaino application internal data directory
 ZAINO_DATA_DIR=/home/zaino/.cache/zaino
 
-# --- Zallet Configuration ---
+# =============================================================================
+# Zallet Configuration
+# =============================================================================
 # Zallet Rust log level
-ZALLET_RUST_LOG=debug
+ZALLET_RUST_LOG=info,hyper_util=warn,reqwest=warn
 # Zallet internal RPC port
 ZALLET_RPC_PORT=28232
 # Zallet host RPC port (for external access to Zallet RPC)

.github/workflows/build-z3-images.yaml

@@ -23,6 +23,24 @@ on:
       - '.github/workflows/build-z3-images.yaml'
 
 jobs:
+  build-zebra:
+    name: Build zebra Docker
+    permissions:
+      contents: 'read'
+      id-token: 'write'
+      packages: 'write'
+    uses: ./.github/workflows/sub-build-docker-image.yaml
+    with:
+      repository: ZcashFoundation/zebra
+      ref: 'main'
+      dockerfile_path: ./docker/Dockerfile
+      dockerfile_target: runtime
+      image_name: zebra
+      no_cache: ${{ inputs.no_cache || false }}
+      rust_backtrace: full
+      rust_lib_backtrace: full
+      rust_log: info
+
   build-zaino:
     name: Build zaino Docker
     permissions:
@@ -31,9 +49,9 @@ jobs:
       packages: 'write'
     uses: ./.github/workflows/sub-build-docker-image.yaml
     with:
-      repository: gustavovalverde/zaino
-      ref: 'imp-dockerfile'
-      dockerfile_path: ./docker/Dockerfile
+      repository: zingolabs/zaino
+      ref: 'dev'
+      dockerfile_path: ./Dockerfile
       dockerfile_target: runtime
       image_name: zaino
       no_cache: ${{ inputs.no_cache || false }}
@@ -49,9 +67,9 @@ jobs:
       packages: 'write'
     uses: ./.github/workflows/sub-build-docker-image.yaml
     with:
-      repository: gustavovalverde/wallet
-      ref: 'feat-add-docker'
-      dockerfile_path: ./docker/Dockerfile
+      repository: zcash/wallet
+      ref: 'main'
+      dockerfile_path: ./Dockerfile
       dockerfile_target: runtime
       image_name: zallet
       no_cache: ${{ inputs.no_cache || false }}

.gitignore

@@ -18,6 +18,7 @@ target/
 
 # Ignore all contents of the 'config' directory recursively
 config/**
+!config/zallet.toml
 
 # Un-ignore the 'tls' subdirectory itself, so we can look inside it
 !config/tls/