Skip to content

README.md

Latest commit

 

History

History
272 lines (208 loc) · 12.3 KB

README.md

File metadata and controls

272 lines (208 loc) · 12.3 KB
Raw

CCAM monitoring stack (Prometheus + Grafana)

Prometheus Grafana Docker Node.js

A turnkey Prometheus + Grafana stack that scrapes the dashboard's GET /api/metrics endpoint and renders four auto-provisioned Grafana dashboards (default home: CCAM — Overview). Use it to watch live sessions, agent states, event throughput, and token burn from the same observability stack as the rest of your infra.

No Homebrew, apt, or global installs required. There is no official prometheus or grafana server package on npm (only client libraries and UI components) — the stack uses npm's own install lifecycle instead: npm run monitoring:install runs postinstall in this folder and pulls official release binaries into monitoring/.bin/ (same pattern as Playwright browsers or Electron).

Supported platforms (npm path)

OS Architectures
macOS Apple Silicon (arm64), Intel (x64)
Linux arm64, amd64 (x64)
Windows x64

Node.js 20+ is the only prerequisite. On Windows, run the npm commands from PowerShell or Command Prompt in the repo root (same as the rest of CCAM).

Grafana login (local default)

Open http://localhost:3000 after monitoring:up or monitoring:docker:up. The admin account is auto-created on first start — no manual signup:

Field Value
Username admin
Password admin

Credentials are defined in grafana.defaults.env and applied by both the npm and Docker paths. CCAM — Overview is the default home dashboard (GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_UID=ccam-overview) with PromQL queries against your live /api/metrics scrape — no sample or synthetic data. The Prometheus datasource is pre-provisioned, so you land straight in the UI after login.

If login fails after an earlier Grafana run, reset local state: rm -rf monitoring/.data/grafana (npm) or recreate the grafana-data Docker volume, then start the stack again.

monitoring/
├── package.json                           # npm install downloads binaries (postinstall)
├── grafana.defaults.env                   # default admin / admin credentials
├── scripts/                               # lifecycle helpers (setup / up / down)
├── prometheus/
│   ├── prometheus-native.yml              # scrape config for npm-managed stack
│   ├── prometheus.yml                     # scrape config for Docker stack
│   ├── prometheus-docker.yml              # all-Docker stack (docker-compose.full.yml)
│   ├── ccam-rules.yml                     # recording rules (derived from live scrapes)
│   └── consoles/index.html                # Prometheus CCAM console (pre-built graphs)
├── grafana/
│   ├── provisioning/…                     # Docker datasource + dashboard provider
│   ├── provisioning-native/…              # npm-managed datasource template
│   └── dashboards/                        # ccam-overview + 3 focused boards
└── docker-compose.yml                     # optional Docker path

Quick start (npm — no Docker)

  1. Start the dashboard on loopback (default npm start on port 4820). No DASHBOARD_ALLOWED_HOSTS override is needed — Prometheus scrapes 127.0.0.1:4820, which the server's Host guard already accepts.

    npm start

  2. One-time install — uses npm's postinstall to download official Prometheus + Grafana OSS binaries into monitoring/.bin/ (~150 MB total):

    npm run monitoring:install

    (monitoring:setup is an alias for the same command.)

  3. Bring up the stack:

    npm run monitoring:up

  4. Open Grafana at http://localhost:3000 (login admin / admin). The CCAM — Overview dashboard is already there — no import step. For Prometheus, open the pre-built CCAM console at http://localhost:9090/consoles/index.html (live graphs + tables), or http://localhost:9090 → Consoles → index.html (Status → Targets: ccam should be UP).

Stop with npm run monitoring:down. For a foreground session with logs on the terminal, use npm run monitoring:start instead (Ctrl+C stops both).

Quick start (Docker / Podman)

Use this when you prefer containers or when the dashboard itself runs in Docker.

  1. Start the dashboard so the container can scrape it. The server's DNS-rebinding guard only accepts loopback Host headers, and Prometheus (in Docker) reaches the host as host.docker.internal — so allow that Host:

    DASHBOARD_ALLOWED_HOSTS=host.docker.internal npm start

    When the dashboard runs inside Docker (docker compose up at the repo root), set the same variable on the agent-monitor service:

    DASHBOARD_ALLOWED_HOSTS=host.docker.internal docker compose up -d

    Without this you'll see the Prometheus target stuck DOWN with 403 EBADHOST. If you also set DASHBOARD_TOKEN, see Auth below.

  2. Bring up the monitoring stack:

    npm run monitoring:docker:up

  3. Open Grafana at http://localhost:3000 (login admin / admin).

Tear down with npm run monitoring:docker:down.

All-in-one Docker (app + monitoring)

Runs the dashboard, Prometheus, and Grafana on one network — no host.docker.internal wiring required:

npm run docker:full:up
npm run monitoring:verify

Stop with npm run docker:full:down.

Deployment options

Dashboard Monitoring Commands
npm (npm start) npm monitoring:install → monitoring:up
npm Docker DASHBOARD_ALLOWED_HOSTS=host.docker.internal npm start → monitoring:docker:up
Docker (docker:up) Docker DASHBOARD_ALLOWED_HOSTS=host.docker.internal docker compose up -d → monitoring:docker:up
Docker npm docker:up + monitoring:up (scrapes host 127.0.0.1:4820)
Docker full stack included docker:full:up

Verify any running stack: npm run monitoring:verify.

npm scripts

Command Description
npm run monitoring:install npm install in monitoring/ — downloads binaries via postinstall
npm run monitoring:setup Alias for monitoring:install
npm run monitoring:up Start Prometheus (:9090) + Grafana (:3000) in the background
npm run monitoring:down Stop the npm-managed stack
npm run monitoring:start Foreground start (Ctrl+C stops both)
npm run monitoring:docker:up docker compose Prometheus + Grafana
npm run monitoring:docker:down Tear down the Docker monitoring stack
npm run monitoring:verify Health-check dashboard + Prometheus + Grafana + scrape target
npm run docker:up Start the dashboard container only
npm run docker:down Stop the dashboard container
npm run docker:full:up Dashboard + Prometheus + Grafana (all Docker)
npm run docker:full:down Tear down the full Docker stack

Bundled Grafana dashboards

Four dashboards are auto-provisioned from grafana/dashboards/ — all query live /api/metrics data (no sample or synthetic series):

Dashboard UID Focus
CCAM — Overview ccam-overview Default home — fleet snapshot, totals, breakdowns, rates
CCAM — Sessions & Agents ccam-sessions-agents Session lifecycle, agent states, WebSocket clients
CCAM — Tokens & Events ccam-tokens-events Cumulative tokens/events, throughput rates, cache efficiency
CCAM — Platform Health ccam-platform Scrape/API uptime, process memory, remote sources, build info

Each board links to the others in the header. After monitoring:up, open http://localhost:3000/dashboards or land on Overview as the home dashboard.

What's on the Overview dashboard

The bundled CCAM — Overview board uses only metrics from your database. Cumulative stat panels (total sessions, events, tokens) show substantial numbers on the first scrape; rate panels need a few minutes of scrape history before lines appear.

Section Example queries
Live fleet ccam_sessions{status="active"}, ccam_agents{status="working"}, ccam_websocket_clients, sum(ccam_sessions)
Database totals ccam_sessions{status="completed"}, ccam_events_total, ccam_tokens_total, sum(ccam_tokens_total)
Breakdown ccam_sessions (pie), ccam_tokens_total (bar gauge)
Over time ccam_sessions, ccam_events_total, ccam_tokens_total
Rates rate(ccam_events_total[5m]), rate(ccam_tokens_total[5m])
Process ccam_process_uptime_seconds, ccam_process_resident_memory_bytes, ccam_build_info

See docs/API.md → Metrics for the full metric list.

Prometheus quick start

Open the CCAM console first — static HTML that queries Prometheus directly (Prometheus 3.x compatible; no deprecated console template libraries):

http://localhost:9090/consoles/index.html

Also reachable from the Prometheus UI menu: Consoles → index.html.

The console runs real PromQL against your scraped metrics: session totals, cumulative events/tokens, working agents, and drill-down links into the Graph UI. If the page warns about a missing scrape target, start CCAM on port 4820 and wait ~15s for the first poll.

Graph tab bookmarks

Open http://localhost:9090 and paste any of these into the Graph tab (all read live scraped data — never seeded):

Query What it shows
sum(ccam_sessions) Total sessions across all statuses
ccam_events_total Cumulative hook events in the database
sum(ccam_tokens_total) All token kinds combined
ccam:sessions:total Same as above via recording rule
ccam:events:rate5m Events per second (5m window)

Pre-filled graph links (bookmark these):

Configuration

  • npm-managed stack. Scrape target lives in prometheus/prometheus-native.yml (default 127.0.0.1:4820). Binary versions are pinned in scripts/paths.js.
  • Docker stack. Host-native dashboard: prometheus/prometheus.yml (host.docker.internal:4820). All-Docker stack: prometheus/prometheus-docker.yml (agent-monitor:4820, used by docker-compose.full.yml).
  • Auth (DASHBOARD_TOKEN). If the server requires a token, uncomment the authorization block in the relevant prometheus config and set credentials to your DASHBOARD_TOKEN. (The DASHBOARD_ALLOWED_HOSTS step is still required for Docker scrapes — the Host guard runs independently of the token.)
  • Scrape interval lives in the prometheus config (global.scrape_interval).
  • Grafana admin password — defaults to admin / admin via grafana.defaults.env (Docker) and grafanaAdminEnv() in scripts/paths.js (npm). Applied on first start only; wipe monitoring/.data/grafana or the grafana-data volume to re-seed.

Security note

/api/metrics exposes aggregate operational counts (session/agent tallies, event and token totals, uptime) — no prompts, transcripts, costs, or secrets. It sits behind the same loopback/Host guard and optional DASHBOARD_TOKEN as the rest of the API; scraping is only possible once you explicitly allow the scraper's Host (and token, if set). Keep Grafana/Prometheus on a trusted network or behind your own reverse proxy.