feat: restore cadvisor + redis-exporter metrics sidecars#2676
Conversation
Re-add the per-container (cadvisor) and redis (redis-exporter) metrics sidecars to the provider compose stack. These were added in #2058 then removed in 6d7c062 because they were mis-indented under redis-stack (so not real services) and redis-exporter lacked REDIS_ADDR. This restores them as correctly-indented top-level services with static IPs, fixes redis-exporter by pointing it at redis://redis-stack:6379, and runs cadvisor privileged with /dev/kmsg per the documented setup. Also mount the host's /proc and /sys into the vector container (with PROCFS_ROOT/SYSFS_ROOT) so the host_metrics source reports the real host rather than the vector container's namespaced view. The vector.toml scrape config for cadvisor:8080 and redis-exporter:9121 already survived the revert, so no vector config changes are needed.
There was a problem hiding this comment.
Pull request overview
This PR restores missing metrics sidecars (cAdvisor + redis-exporter) to the provider Docker Compose stack and adjusts the vector container so its host_metrics source can read the host’s /proc and /sys instead of the container’s namespaced views.
Changes:
- Add
redis-exporterservice (internal-only exposure on:9121) configured to scraperedis-stack. - Add
cadvisorservice (internal-only exposure on:8080) with the required host mounts for container metrics. - Update
vectorservice to mount host/procand/sysand setPROCFS_ROOT/SYSFS_ROOT.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| environment: | ||
| # The exporter connects to redis over the internal network. Without an | ||
| # explicit REDIS_ADDR it defaults to its own localhost:6379 and scrapes | ||
| # nothing. REDIS_PASSWORD is filled from the deploy-exported provider env | ||
| # (same value as REDIS_CONNECTION_PASSWORD); empty means no-auth redis. | ||
| - REDIS_ADDR=redis://redis-stack:6379 | ||
| - REDIS_PASSWORD=${REDIS_CONNECTION_PASSWORD:-} | ||
| env_file: |
There was a problem hiding this comment.
Correct, and this is a known constraint — `${REDIS_CONNECTION_PASSWORD:-}` resolves at compose-render time from the shell/top-level `.env`, not from `env_file`. It is called out explicitly in the PR description as a deploy note: the var must be exported into the deploy shell at `compose up` (the same way `HOST_IPV4` already is) for a password-protected Redis to authenticate. The deploy/ansible that renders and exports the provider env lives outside this repo, so the right fix is on that side rather than hardcoding here. Leaving the `:-` default so a no-auth Redis still works. @goastler to confirm the export against the deploy tooling.
Add two more metrics sidecars to the provider stack and wire them into vector's remote-write pipeline: - mongodb-exporter (percona/mongodb_exporter): richer mongo metrics (replication, per-collection/index stats, query exec, connection pool) on :9216, beyond the basic server status from vector's built-in mongodb_metrics source. URI passed via env, not --mongodb.uri, to keep the password out of the container's process args. - smartctl-exporter: disk SMART health (reallocated sectors, temperature, wear, predicted failure) on :9633, not captured by host_metrics/cadvisor. vector.toml gains prometheus_scrape sources for both and includes them in both OpenObserve remote-write sinks. Also address review feedback: switch redis-exporter depends_on to the list form (no conditional depends_on elsewhere in the repo's compose).
What
Restore and expand the metrics-sidecar suite for the provider compose stack, and fix vector's host-metrics collection.
Why
cAdvisor + redis-exporter were added in #2058 then removed in
6d7c062449("remove invalid properties") because they were mis-indented underredis-stack(so docker compose treated them as keys of that service, not real services), andredis-exporterlackedREDIS_ADDR. Thevector.tomlscrape config survived the revert — only the containers were missing. This PR restores them correctly and adds two more exporters for fuller coverage.Sidecars (
docker/docker-compose.provider.yml)gcr.io/cadvisor/cadvisoroliver006/redis_exporterpercona/mongodb_exporterprometheuscommunity/smartctl-exporterAll are correctly-indented top-level services with static internal IPs (
172.18.0.11–.14),production/stagingprofiles, and json-file logging caps.privileged+/dev/kmsgper the documented setup.REDIS_ADDR=redis://redis-stack:6379.MONGODB_URIvia env (not--mongodb.uri) so the password isn't in the container's process args;--collect-all --compatible-mode. Complements vector's existing built-inmongodb_metrics(basic server status).privilegedto issue SMART commands to host disks.vector
/proc→/host/procand/sys→/host/sys(read-only) +PROCFS_ROOT/SYSFS_ROOT, sohost_metricsreports the real host rather than the vector container's namespaced view.vector.toml: newprometheus_scrapesources for mongodb-exporter (:9216) and smartctl (:9633), each tagged with host/env and included in both OpenObserve remote-write sinks.Validation
docker compose --profile production configrenders cleanly (EXIT=0) andvector.tomlparses as valid TOML with all sources wired into both sinks.Deploy note
redis-exporter'sREDIS_PASSWORD=${REDIS_CONNECTION_PASSWORD:-}(andmongodb-exporter'sMONGODB_URIcreds) rely on those vars being exported into the deploy shell atcompose up(likeHOST_IPV4), since compose${...}interpolation doesn't readenv_file. If prod redis usesrequirepassand the var isn't shell-exported, the exporter will fail auth. Worth confirming against how the provider env is rendered/exported by the deploy tooling.