Observability — logs and metrics

bex makes a running platform observable without kubectl: an operator or an AI agent debugging a deploy reaches App logs and metrics through the same bearer-authed bex-api it uses for lifecycle verbs. This is GOAL.md #2 ("basic obs for operation") and the AI-native pillar in ADR008-vision.md — agents can't fix a failing deploy they can't read.

Logs shipped first: highest operational value, and the simplest backend (pod logs, no metrics-server dependency). Metrics follow, over the same one-Core-many-adapters shape.

Logs

One Core logs read, three adapters — the bex-api invariant. The MCP list_logs tool is the agent surface; REST and GraphQL expose the same read to the public API and dashboard.

flowchart LR
  rest["REST GET /v1/logs<br/>GET /v1/logs/subscribe"] --> core
  gql["GraphQL logs(...)"] --> core
  mcp["MCP list_logs"] --> core
  core["Core.Logs / QueryLogs / FollowLogs"] --> hist["LogHistorySource<br/>(BEX_LOKI_URL)"]
  core -.fallback.-> src["PodLogSource"]
  core --> stream["PodLogStream (tail)"]
  hist --> loki["Loki ← log-shipper DaemonSet"]
  src --> pods["App pods (app.bex.co/app label)"]
  stream --> pods
  loki --> pods

Core.Logs(name, tail) — tail-N aggregation across replicas; the unfiltered convenience read. Returns LogEntry{timestamp, message, labels} (labels service/instance/container).
Core.QueryLogs(LogQuery) — adds Render's filters and paging; the read path all three adapters (REST, GraphQL, MCP list_logs) go through. A dpg-… resource dispatches to the Database-authorized Postgres path (w3/m28); a red-… resource dispatches to the KeyValue-authorized Valkey path (w3/m30). Both preserve the same public operation.
Core.FollowLogs(LogQuery, emit) — live tail; the SSE stream.

Two backends sit behind the read verbs, each an injected source so the domain stays clientset/HTTP-free (like PodLogSource, both are faked in tests with no cluster):

LogHistorySource (NewLokiSource, gated by BEX_LOKI_URL) — the durable backend Core.QueryLogs/Logs prefer when wired. It translates the authorized LogQuery into a LogQL query_range: {namespace, app} for services or {namespace, database} for managed Postgres, plus text, time, instance, and limit. Alloy feeds both sources. This is what makes logs survive a pod restart: pre-restart App and CNPG lines remain searchable.
PodLogSource (NewPodLogSource) — the live fallback for QueryLogs/Logs when BEX_LOKI_URL is unset: reads the kubelet's pods/log ring buffer directly (the one subresource controller-runtime's client can't serve). Apps select app.bex.co/app and container app; Postgres selects exact cnpg.io/cluster=<dpg-id> and container postgres; Key Value selects app.bex.co/keyvalue=<red-id> and container valkey. No history — a restart loses the buffer — but zero extra infrastructure.

PodLogSource/PodLogStream live in podlogs.go; LogHistorySource in loki.go. All three are injected in cmd/api/main.go.

The tail reads pod logs, not Loki

Core.FollowLogs (the GET /v1/logs/subscribe SSE stream) always reads PodLogStream (pod logs), never Loki — even when Loki is wired. The default/App tail follows the App containers; an explicit type=build follows the newest running build pod in BEX_BUILD_NAMESPACE (BuildKit, signed BuildKit, or kpack, with running container names discovered from Pod status). The tail is for new lines going forward, where following the kubelet stream directly is real-time with zero ingest lag, adds no moving parts, and — critically — does not die when Loki is down: history degrades to the live buffer, the tail is unaffected. Loki's own tail endpoint would give a single history+tail source at the cost of a small ingest lag and a new failure mode on the live path; the durability win is on the historical query (QueryLogs), which Loki already owns, so the tail stays on pods. The one accepted consequence: a freshly-restarted pod's tail starts from that pod's new buffer — but the pre-restart lines are served by QueryLogs from Loki, so nothing is actually lost.

Request lines get a second producer (w4/m237, 2026-10-19). The rule above holds for app and build lines, but type=request has no pod source — the access line is written by Traefik, not the tenant's process — so a pod-only tail could never stream it, and the dashboard's "Live" request lines arrived on the history refetch (~30 s). The subscribe stream now also runs a store poll: every RequestPollInterval (2 s default) it reads request lines that reached Loki since an exactly-once cursor (newest emitted instant + the identities emitted at it, so a shared-timestamp group is neither dropped nor repeated) and merges them into the same stream. App/build lines keep zero ingest lag and never depend on Loki; request lines carry shipper + one poll interval of lag, and their structured filters are honored by the store query, forwarded from the subscription. A request-only subscription with no store wired fails closed (503) before any header; mixed streams without a store still serve their pod lines. Refusals that remain: store-only filters over app/build lines (labels exist only in the store) and producerless types — named 400s before a subscription slot is taken.

Durability & retention

Loki keeps 21 days of searchable history (limits_config.retention_period: 504h, compactor-enforced) — enough for the ADR055 registry/static identity migration's 14-day clean window plus an evaluation buffer (w2/m93, runbooks/registry-static-identity-migration.md Phase 4). Render's searchable window is tiered by plan — Hobby 7 days, Pro 14, Scale/Enterprise 30 — so 21 days sits inside the Pro/Scale range; lowering back toward the Hobby floor is one knob (retention_period in deploy/gitops/base/loki.yaml, plus the PVC size) once Phase 4 closes. Loki runs single-binary on a filesystem PVC (no object store) — the same posture as Prometheus's PV and the etcd/OpenBao Raft volumes: history survives a pod restart because it's on a real PV. On prod the PVC rides the cluster's default hcloud-volumes StorageClass (a Hetzner Cloud Volume — a network volume that survives a node rebuild/loss), so a node going away does not lose the history; the disposable CAPD mock uses local-path (node-local), where a node loss does lose it. Cross-host log durability beyond one volume's lifecycle would mean an object-store backend, deferred like the others. The local overlay shrinks the PVC to CAPD scale (2Gi) since the mock cluster is disposable. The same durability covers tenant-node replacement (autoscaler scale-down/up churn), not just a crash: the log-shipper DaemonSet tolerates every taint so it ships from any node the autoscaler adds, and the hcloud volume follows Loki's pod to whichever node it's rescheduled onto — no further work needed.

Cluster enablement

deploy/gitops/base/loki.yaml runs the one Loki (single-binary, filesystem PVC, 21-day retention) and deploy/gitops/base/log-shipper.yaml runs the Grafana Alloy DaemonSet that ships App (type=app), request (type=request), build, managed-Postgres (type=postgres), managed-Valkey Key Value (type=keyvalue), and platform streams (type=platform: dashboard UI service=dashboard — w4/m88; Zot registry service=zot in bex-registry and static-server service=static-server in bex-system — w2/m93 migration clean-window evidence; bex-api service=bex-api and the operator manager service=operator in bex-system — w7/m157, so incident evidence survives replica replacement). Platform budget (w7/m157): the two core services add ≈10 MB/day raw (operator zap console lines, severity parsed from the level column; bex-api's few std-log lines stay level=unknown) to ≈31 MB/day across all streams measured 2026-09-28, against a 20 GiB Loki volume then 1.9 GB used; retention is the shared 21-day compactor window. The selection is a closed allowlist — activator, SNI proxies, egress meter and ssh-gateway stay out, and no new label carries a tenant id. The Postgres pipeline requires the operator-stamped app.bex.co/component=database marker, keeps only the postgres container, and labels the stream with the immutable CNPG cluster/Database id; platform CNPG clusters are excluded. The Key Value pipeline selects pods whose app.bex.co/keyvalue label is non-empty (operator-stamped tenant Valkey pods only), keeps only the valkey container, and labels the stream with the keyvalue=<red-id> label. Platform pod logs are selected by namespace / component alone (no fake app.bex.co/app); they are operator-facing and are not part of bex-api's tenant log API. scripts/registry-migration-readiness.sh queries the zot/static-server streams for legacy dual-read hits. BEX_LOKI_URL=http://loki.monitoring.svc:3100 points bex-api at it and flips historical reads from pod buffers to durable history.

Edge refusals remain request logs (w8/m115, 2026-10-18). A service inbound IP rule's Traefik 403 is retained in that App's request history even though the middleware never calls its backend (OriginStatus=0, no ServiceName). The shipper recognizes access lines by DownstreamStatus and attributes them from Traefik's trusted ingress name and anchored tenant/default namespace, retaining the older service-name fallback. Client-supplied RequestHost never selects a tenant stream. The existing status, path, method and host filters, unfiltered history, and status-value discovery expose these refusals consistently through REST, GraphQL, MCP, the CLI and the dashboard, matching Render's documented behavior. This adds no labels or API fields and makes no claim about rejections that occur before Traefik.

Tenant log content (w8/m48, 2026-10-02)

The Postgres shipper and direct-pod fallback classify a CNPG line by one allow-list (w5/m122). A logger=postgres, msg=record entry with a record object is unwrapped into PostgreSQL message/severity fields. A line that is not a JSON object passes unchanged. Every other JSON object is the instance manager's (named loggers, non-record PostgreSQL chatter, probes, its net/http server's TLS errors) and is dropped. Both read the line as the shipper's JSON stage does, with exact keys and any value types, and render a field CNPG omitted as nothing. Both sides are tested against one table, lego/backend/internal/logs/testdata/tenant-log-lines.json: the shipper harness (scripts/test_log_shipper.py) runs it through the real pipelines, and the Go direct reads through theirs, so the two cannot drift. Temporary classification labels are removed before Loki ingestion. REST, GraphQL, MCP, the CLI, and the dashboard read these same sources; the Render logs API likewise exposes Postgres resources and severity filtering. This is bex's CNPG normalization, not a claim that Render uses the same underlying log format.

App pods tail CRI files (w4/m174, 2026-10-05). Tenant App containers ship from the kubelet's on-disk CRI files (loki.source.file + stage.cri), as build and pre-deploy pods have since w6/m123, not through loki.source.kubernetes. The pinned API tailer skips any line not stamped strictly after the previous one (kubetail/tailer.go:277), and the CRI file is not in timestamp order: containerd copies stdout and stderr on separate goroutines that stamp each line when they read it. On kind, 4 of 20 runs of a five-line fast cron command left a line (nearly always the stderr one) written after a later-stamped line, and production lost a cron run's error: line in 3 of 6 runs. The file tailer reads by byte offset, so every line ships with its own write timestamp. The filename and stream labels the file source and stage.cri add are dropped, keeping the App stream labels unchanged. Positions live on the node (/var/lib/bex-log-shipper, mounted as Alloy's storage path), so a restarted replica resumes each file instead of re-reading it. The kubelet's answers once a container's log is gone (unable to retrieve container logs …, failed to try resolving symlinks …) never appear in the file, so the App pipeline no longer carries their drops. Before w4/m174 they were dropped there by their text, as the Postgres pipeline did until w5/m122. The direct pod-log read still re-types either body as the platform "logs for this instance are no longer available" line. One-time rollout effect: the first file-tailing replica on each node reads current container files from the start. Loki rejects lines more than its out-of-order window behind a stream's newest entry. Lines inside that window were shipped by the API tailer with a trailing newline that the file path does not carry, so they are not byte-identical and can appear twice in history once.

The file source must not lose lines to pod churn (w4/m174 t007, 2026-10-18). After the switch, production still lost about one line in six fast cron runs, a different line each time. scripts/log-shipper-kind-replay.py replays the production Loki and log-shipper charts on a throwaway kind cluster with the DoD command as Jobs. On Alloy v1.11.2 it lost 4 of 800 lines. Each lost line was read from its file (loki_source_file_read_lines_total) and never sent (loki_write_sent_entries_total), with nothing counted as dropped. The cause is v1.11.2's loki.source.file: it re-applies readers on every target change, which every pod churns, through a temporary drain goroutine that it then cancels, and an entry held mid-send is dropped. v1.11.3 forwards from one dedicated consumer and applies tasks separately, and lost 0 of 800. The log-shipper image is therefore built on v1.11.3 (deploy/log-shipper/Dockerfile), and the published digest also lost 0 of 800. Its --probe delayed mode, a321's begin marker, a 90-second wait and an end marker, kept 160/160 lines on both versions, which is too small a sample to confirm or rule out that single prefix loss. The replay is the check for any future Alloy bump.

Datastore pods tail CRI files too (w5/m122). The Postgres and Key Value pipelines moved to loki.source.file + stage.cri as well, so every tenant pipeline reads only what its containers wrote. The kubelet's answers once a container's log is gone come from its log API, never a file, and no tenant pipeline carries a message-text filter for them any more; the harness asserts that the app, build, Postgres and Key Value pipelines have file sources only. Every stage.cri caps a split line's reassembly at 200000 bytes, under Loki's 256 KiB limit, because rebuilding an unbounded one grows with its square and stalls the node's pipelines. The direct reads apply the same rule by source: every container line carries the timestamp the kubelet stamps with Timestamps: true, and its own answers carry none. An App read re-types an untimestamped line as the platform "logs for this instance are no longer available" line. A datastore read drops it, through the logs service and the datastore adapters' own fallback alike (datastorelogs.ParseLine). A follow is the one exception: it starts at its since second, and the kubelet stamps nothing on the rest of a split line whose first chunk it skipped as older. So a follow holds an unstamped first line, which is the kubelet's answer when the stream ends there and the tenant's text when more output follows. The rollout has the same one-time effect as w4/m174's: the first file-tailing replica on a node reads current Postgres and Key Value container files from the start. Lines inside Loki's out-of-order window can appear twice in history once, and the older lines Loki rejects raise the LogDeliveryLokiRejected and LogDeliveryEntriesDropped warnings once per node. Existing wrapped or placeholder lines already stored in Loki are unchanged and age out under normal retention.

A reassembled CRI record keeps its first chunk's timestamp (w4/m199). The runtime writes a long line, or one written in pieces with a delay, as partial (P) chunks completed by an F chunk, each stamped when it was written. The kubelet's pod-log API, and so the live tail, stamps the logical record with its first chunk's time. Upstream stage.cri (unfixed through v1.19.2) kept the last chunk's time on both merge branches, so a 50,075-byte record written in two pieces 120 ms apart reached history 120 ms after its live copy. The ids differed and the dashboard rendered it twice. The log-shipper therefore runs ghcr.io/bex-co/bex-alloy: Alloy v1.11.3 (v1.11.2 until w4/m174 t007) with one carried patch keeping the first chunk's time and extracted time on a repeated P and on the completing F. The recipe is deploy/log-shipper/ (pinned upstream commit, patch with its own regression test, official v1.11.3 image with only /bin/alloy swapped), published by .github/workflows/log-shipper-image.yml and pinned by digest in deploy/gitops/base/log-shipper.yaml. Message assembly, buffer keys (stream labels, so stdout and stderr stay apart), the 100-buffer count bound and the 200000-byte cap are unchanged, and a single F keeps its own time. scripts/test_log_shipper.py replays all four CRI pipelines (app, build, Postgres with CNPG unwrapping, Key Value) in the pinned image; BEX_ALLOY_IMAGE replays a candidate build before it is pinned. Records already stored with the last chunk's time keep it and age out under retention.

The pod-log producer keeps stdout and stderr records apart (w4/m200). The live tail reads one combined Timestamps:true pod-log response. The stock kubelet reader (k8s.io/cri-client ReadLogs) wrote each CRI partial chunk as it read it and tracked line starts with one flag shared by both streams. A stderr line written between a long stdout record's chunks therefore came out joined to the stdout prefix (32,768 + 89 bytes), followed by an orphan suffix. History was correct, and the dashboard showed the corrupt row beside it. No bex-api or frontend repair can restore the boundary once the response has erased it. Selecting one stream (alpha, and TailLines must then be nil) shares the same flag.

The node image therefore carries a patched kubelet (infra/packer/log-record-framing/, see infra/packer/README.md). It assembles each stream's record and writes it whole, stamped with its first chunk, as m199's collector does for history. It holds at most 1 MiB per stream, after which a longer record streams on as before. It flushes an unterminated record when a finite read or an exited container's follow ends. It skips a record whose first chunk is older than sinceTime, rather than emitting its unstamped rest, so the follow reader's held-unstamped-line rule now only meets the kubelet's own answers.

A TailLines start inside a split record still begins with that record's remaining chunks, which is the physical-line tail semantics. Live and Loki independence are unchanged. Proven on a real kubelet's pod API (kind, the same file read before and after the swap); production nodes get it only through a reviewed snapshot bake and roll.

Log labels (and the cardinality budget)

A Loki label is a stream, so only bounded fields become labels; unbounded ones stay in the line and are filtered at query time. That single rule is the whole taxonomy:

field	where it lives	why
`namespace`, `app` or `database` or `service`	label	the scope of every query — equality matchers resolved after App/Database authorization for tenant streams; `service=dashboard` is the bounded platform-UI key (w4/m88), never a caller-controlled selector
`type` (`app`/`request`/`build`/`postgres`/`platform`)	label	bounded source vocabulary
`pod`, `container`	label (app/platform logs)	bounded by replica count; store `pod` projects to Render's public `instance` id (name-derived; see ADR035 / w5/m87)
`level`	label (app/platform logs)	5 values, hard-capped by the shipper's normalizer (below)
`method`	label (request logs)	the HTTP verb set (≤8)
`status`	label (request logs)	the status codes an App actually returns (≤~15); Render's `statusCode`
`path`, `host`	line only	unbounded per request — one label per URL would mint a stream per URL

Worst case ≈ (replicas × levels) + (methods × statuses) streams per App — low hundreds; typically under 20. path/host are still fully filterable: the access line is JSON, so a query parses them out with LogQL's | json stage (request_path/request_host) instead of indexing them. Promoting path to a label would be a cardinality incident — TestPathAndHostNeverBecomeLabels guards it.

Request logs (type=request) are Traefik's JSON access log (logs.access in deploy/gitops/base/values/traefik.values.yaml), attributed back to the App by the access line's ServiceName — Traefik names an Ingress-backed service <namespace>-<service>-<port>@kubernetes, and the operator names the k8s Service after the App CR, so the shipper reconstructs the {namespace, app} labels bex-api's LogQL selector queries by (namespace = app.Namespace, app = app.Name) from that name. Tenant-namespace fix (w6/m131): unlike the app/build/postgres pipelines — which read the namespace straight from pod metadata — the request pipeline is the one that parses it out of ServiceName, and its regex was anchored to the literal default. Under ADR043 a tenant App lives in its workspace's own namespace (tea-<xid>) and its CR name is itself tenant-prefixed, so every real ServiceName is tea-<xid>-tea-<xid>-<app>-<port>@kubernetes — which the default-anchored regex never matched, so every tenant access line was dropped as not_a_tenant_app and type=request was silently empty for every service in production (found live, 69th /qa-find-bugs run). The regex now matches a tea-<xid> tenant namespace as well as the shared/storeless default; a mislabel could only ever hide a line, never surface another tenant's, because bex-api pins both namespace and app to the caller's own resolved values. A line the regex can't attribute is dropped, not guessed, except for an explicit host allowlist (w4/m88): RequestHost=dashboard.bex.co is retained under the bounded labels namespace=dashboard + service=dashboard (still type=request); host itself stays line-only. Other non-App edge hosts (bex-api, oauth, …) remain dropped until explicitly allowlisted the same way. Request headers are dropped at the source: they carry Authorization/Cookie, and a request log is not a place to leak a credential. So are four platform-internal fields (w8/047): RequestCount (Traefik's process-wide counter, which would let any tenant sample total edge traffic), ServiceAddr/ServiceURL (pod IPs) and RouterName (internal routing names). The message stays the raw JSON access line (every kept field intact and searchable) rather than a prettified summary — a divergence from Render's rendered request line, in favor of losing nothing a tenant may see. The kept fields are an allowlist (w5/076): accessLog.fields.defaultMode: drop with every field tenants already saw named keep, so a field a Traefik upgrade adds stays out of tenant logs until it is reviewed and listed. access_log_fields_test.go pins the list and fails when a field the shipper or bex-api reads is not kept. The chart-rendered flags were checked against Traefik v3.7.5 itself: the same request logs the same fields before and after.

level on app logs is parsed, never guessed. The shipper parses JSON and logfmt app lines and promotes their level (or severity) field, normalizing the spellings (err/fatal/panic/critical → error, warn → warning, notice → info, …) into the five buckets error|warning|info|debug|unknown. warning is Render's name, the value the pinned CLI's --level sends. Before w8/031 the bucket was warn, and a level=warning query matched nothing. The query side maps every Render level name onto the stored buckets (warning → warning|warn, so streams stored before the change stay findable within retention; notice → info; critical/alert/emergency → error). unknown stays: Render has no such value, but labelling an unclassified line as a guessed severity would be worse (decision recorded in w8/031). This includes Go's slog.NewTextHandler output (time=… level=ERROR msg="request failed"). A line without a recognized structured severity is labelled unknown — the honest answer, because bex does not know. Substring matching is deliberately not used: plain ERROR … and msg="level=error" remain unknown. bex recommends structured logging; it does not require it. scripts/test_log_shipper.py renders the locked Helm chart and runs its actual app pipeline in the chart's Alloy image, asserting JSON/logfmt extraction, normalization, message-only exclusions, and the five-value cardinality budget.

Log filters

Historical line queries use Render's logs filter vocabulary, with the supported matching behavior and source limits below:

filter	mechanism	notes
`type`	stream selection	`app` ∪ `request` ∪ `build`; default (no filter) = `app` ∪ `request`
`level`	label matcher	App and managed-Postgres logs; `unknown` is a real, queryable App value
`instance`	label matcher (`pod`) after translating the public instance id (or legacy raw-name bookmark) back to an authorized pod name	app logs — a request line comes from the edge, not a replica
`statusCode`	label matcher (`status`)	exact (`404`) or class (`4xx`)
`method`	label matcher	request logs
`path`, `host`	`\| json` line filter	request logs; unbounded, so never labels (above)
`text`	line filter	case-insensitive literal substrings; OR across all supplied values
`textMode`	Bex extension	`raw` (default) matches stored bytes; `display` matches the terminal-display text (below)
`startTime`/`endTime`	query range
`direction`	`backward` (default) / `forward`	which end of the window `limit` keeps, and the page's order: backward lists the most recent first, forward the oldest first (Render's List logs contract, w8/m116). `hasMore` says whether a line lies beyond the page: each page reads one line past `limit`, and when a shared-timestamp group kept whole absorbs that line, one more past the page, so a cursor walk does not land on an empty page. The bex mobile app and the dashboard order what they show by time, not by page order
`limit`	line cap	default 20, max 100 (Render's paging range)
`task`, `taskRun`, `sandbox`	refused	a named 400 on `/logs`, `/logs/values` and `/logs/subscribe`: Render's Workflows task-run and sandbox streams have no bex counterpart (ADR018 omission, DO_NOT_DO #18). They used to pass the strict-query gate and be ignored, which widened the read to every log (w8/132); `TestEveryDeclaredLogFilterIsParsedOrRefused` keeps every declared filter parsed or refused

Managed Postgres uses the shared time/text/direction/limit behavior plus instance and level; service/request-only filters are named 400s. Severity filtering uses the same Render aliases and OR/* wildcard semantics as App history, so level=error includes normalized PostgreSQL ERROR/FATAL/PANIC records. The CNPG pod-buffer fallback applies the same severity match to its bounded raw tail before capping matching results; it does not recover older or restarted-pod history. Level discovery uses Loki and returns ErrLogStoreUnavailable when that source is absent. Instance discovery is scoped to {namespace, database} in Loki and falls back to the already authorized Database's exact CNPG pod set. See the captured contract and attribution rules. Managed Key Value (Valkey) supports time/text/direction/limit/instance, with no severity filter; instance discovery is scoped to {namespace, keyvalue} in Loki and falls back to pods matching the app.bex.co/keyvalue=<red-id> label. Live-tail (FollowLogs) returns a named 400 for Key Value resources — there is no persistent Valkey connection-scoped log stream to tail.

Multiple values for one line-query filter OR together; different filters AND. For example, text=errorMarker&text=infoMarker returns lines containing either marker while retaining the resource, time, instance and other supported filters. A line matching both terms is still returned once. Reversing the terms or repeating them does not change the result set or paging behavior.

Text terms (w2/m167). REST retains every text occurrence and MCP list_logs retains the full text array. The shared query drops exact empty strings, lowercases and deduplicates the remaining terms, and treats an omitted, empty or all-empty set as no text filter. It does not trim whitespace: " error " searches for those spaces too. Each term is a literal substring, including *, ., |, quotes and backslashes; neither commas nor regex syntax are interpreted by the server. A scalar GraphQL/dashboard search remains one term, so text: "error,info" searches for that literal comma-containing string. Datastore convenience reads likewise keep their scalar text contract. The same term set reaches App, Postgres and Key Value history, bounded pod-buffer fallbacks, the explicit pre-deploy pod read, App/build followers, and platform build-progress lines. This does not add unsupported sources or filters.

The Render CLI reference documents comma-separated --text values, and the logs API defines text as an array. At Bex's exact Render CLI v2.27.0 pin (a764810a768202704e7206eb7b87a47211fcd98e), cmd/logs.go uses StringSlice; logloader.go preserves the array, and the generated client emits repeated query keys. Thus bex logs --text errorMarker,infoMarker supplies two terms without a launcher rewrite. These sources establish the array/wire contract; OR within text and AND across filters is Bex's contract above, not a claim of a live Render-server comparison.

Display text search (w4/m189, a Bex extension). Log lines are byte-transparent, so a colored message stores its ANSI escapes (QA_\u001b[31mred\u001b[0m_end) and the raw text match cannot find the visible QA_red_end — the escape breaks the substring. The optional textMode=display (REST/subscribe query parameter, GraphQL logs/logLabelValues argument, MCP list_logs/list_log_label_values field) instead matches the same case-insensitive literal terms against the text a terminal shows: displayText (internal/logs/display.go) mirrors the dashboard's stripAnsi (dashboard/src/features/logs/lib/ansi.ts) — CSI/SGR dropped, erase-in-line applied, OSC envelopes and other escapes swallowed, malformed tails consumed, and carriage returns overwrite so hidden text never matches. Returned messages stay raw in every mode, so colors render and ids/hashes are unchanged. Omitted or raw keeps the exact pre-existing semantics, responses and errors; any other value is a named 400. Loki cannot evaluate the projection, so a display search sends only a superset prefilter (any term, or any line carrying ESC or CR — a line without either displays exactly as stored) and reads raw pages of up to 1000 lines inside the same 18 s budget, matching exactly in Go. Coverage advances by raw lines read, not matches: when a raw page fills, coverage stops at its far timestamp and that timestamp group is re-read whole, the caller's inclusive endTime is applied once so a full page on that instant still progresses, and a match behind thousands of unmatched lines is reachable and a budget-exhausted scan pages on (hasMore + cursor) rather than claiming unscanned history is empty. The pod-buffer fallback, followers and build progress apply the same predicate. The dashboard opts in only on the service Logs tab (history and live tail); the deploy panel already matches stripAnsi client-side, and the datastore viewer and build follower keep raw defaults.

Record identity in the readers (w4/m217). GraphQL LogEntry.id is REST's per-record id (instance, original timestamp, message hash). The dashboard keys every identified line by that id, whatever its timestamp precision, so distinct records written inside one millisecond stay distinct. This includes a record stamped exactly on the millisecond, which RFC3339Nano writes with three digits (a872). Before, twenty such records collapsed into one row, keyed by millisecond, instance and message. Live and history canonicalize the stamp before deriving the id, so one record always carries one id. Only an ID-less line keeps the millisecond-bucket key. As a fallback for those, LogIdentity treats the first ID-less copy and the first identified record in a bucket as one record, in either order, so the w9/053 precision-drifted reconnect replay still draws once. Two different ids are never paired. History, older pages, deploy logs, the live ring and the history/live merge all share it. LogEntry stays an embedded cache object (keyFields: false). The field shipped to bex-api before the dashboard selected it, so no dashboard queries a backend that lacks it.

Display changes keep the reader's line (w4/m228). The shared virtualized log list (service, deploy and datastore readers) keeps the reader's logical line when wrapping or timestamps are toggled: it saves the top row's key and offset (or the pinned end), drops and remeasures row sizes, turns off the virtualizer's own size-change scroll compensation and native overflow anchoring while it settles, and re-places that row by key on each measurement pass with the offset clamped to the row's new height. Settling ends after a quiet 250 ms, at most 1 s after the toggle, or on reader input (including Jump to latest); a near-end row clamps to the reachable range without padding.

One case fold, server and deploy panel (w4/m216). The server lowercases terms and messages with Go's strings.ToLower, which maps one rune at a time (unicode.ToLower). The deploy log panel filters lines it has already loaded in the browser, where JavaScript's toLowerCase behaves differently: it folds a word-final Σ to ς, and İ to two code points. So οσ missed an uppercase ΟΣ line, and ος wrongly matched it. The panel now folds through goLowercase, a compact runs table of Go's mapping. lego/backend/internal/logs/testdata/lowercase-runs.json is pinned to unicode.ToLower over every scalar value by a Go test (regenerate with GOWORK=off BEX_UPDATE_LOWERCASE_RUNS=1, the toolchain bex-api is built with, after a Go upgrade), and the dashboard's copy is pinned to that file. Other dashboard searches and every server contract are unchanged.

Render also documents wildcard and regex searches. Bex text search is literal, while structured level/status/method and request path/host matching supports the * wildcard subset. Other metacharacters stay literal. Loki's text predicate escapes each term before joining the alternatives, and every interpolated value is escaped (%q + regexp.QuoteMeta), so a term cannot inject a selector or pipeline stage.

Source limits and refusals:

App pod-log fallback mode (BEX_LOKI_URL unset): the labels live in the store, not in a pod's stdout — so type=request, type=build, and the level/statusCode/method/path/host filters return 503 (ErrLogStoreUnavailable), rather than quietly answering a narrow question with unfiltered lines. type=app, text, instance, time and direction still work, unchanged.
The live tail reads pod logs by design (see above). It supports the App tail and an explicit, standalone type=build, including every text term; it refuses Postgres/Key Value tails, request/pre-deploy tails, mixed build types, and store-only filters. Admission errors are HTTP errors; after the stream opens, errors use the selected transport's terminal error frame (SSE uses event: error). A build subscription with no active or pending build ends with the named no running build is available to follow error. Several resource values are all followed and merged onto the one stream in arrival order (w8/024; it used to follow only the first). Each resource is authorized and holds its own subscription slot before the stream opens, and the first follower to fail ends the tail with its refusal.
An unknown type, direction or label is a 400 naming the value — never a silently widened query.

Static-site request attribution (w4/m113). A static site's ServiceName names the per-App ExternalName alias the operator points at the shared static server — <ns>-bex-static-<app>-<port>@kubernetes (production ground truth, 2026-09-19: "ServiceName":"default-bex-static-hello-static-8080@kubernetes"). The attribution regex therefore recovered app=bex-static-<app>, a name no App has, so every static access line shipped with a label bex-api's selector could never match and type=request was empty for every static site — with bandwidth for the same traffic correct, which is what made it look like a metrics bug rather than a logging one. Traefik already puts the answer on the line: KubernetesIngressName is the App CR name (the operator names each App's Ingress after the App), for compute services too — verified the same day against both shapes. The pipeline now prefers it and keeps the ServiceName regex as the fallback, gated on that regex having matched so a platform edge line (which also carries an Ingress name) still falls to the bounded host allowlist instead of the tenant branch. Both captured production lines are pinned verbatim in shipper_attribution_test.go.

But "the store is up and this stream is not being produced" is not one of those refusals (w6/m131). The rules above all key on the store being absent. When the store is present and healthy but a stream is silently empty — as type=request was for every tenant service until the attribution fix above — the query takes the success path and returns 200 {"logs":[],"hasMore":false}, byte-identical to a genuinely quiet service. That is deliberate, and it is a limit rather than an oversight: bex cannot distinguish the two from a single resource's vantage point. Loki's label index is empty in both cases, so there is nothing to key on, and inventing an error would misreport every quiet service. The honest empty stays.

What closes the gap is out-of-band and platform-wide, where the distinction is decidable — across every tenant app, over hours, zero request lines means the pipeline, not the traffic:

scripts/request-logs-liveness.sh reads Loki's label index every 6h from .github/workflows/ssh-edge-liveness.yml and fails loudly (opening a tracking issue) when a tenant log pipeline has no stream for any tea-<xid> tenant namespace. The tenant qualifier is the whole point: scripts/logs-verify.sh exercises APP_NS=default, which is exactly the namespace the broken regex still matched, so it would have passed green throughout the outage. It now also asserts that the live deployed ConfigMap's ServiceName regex attributes a tenant namespace — the check that actually answers "did the GitOps change reach the running shipper?".
Both tenant pipelines, one probe, two issues (w3/m83 t002). STREAM_TYPES defaults to "request app", so the same label-index read now also asserts type=app — the pipeline w3/m36 found empty for every tenant, the identical silent zero on the other side. The types stay separate all the way to the tracking issue (request-logs-down vs app-logs-down) because their remediations are unrelated: a dark request stream points at the ServiceName attribution regex, while the app pipeline reads its namespace straight from pod metadata, so the suspect there is the Alloy DaemonSet's reach and its app.bex.co/app selector. The script emits one RESULT type=<t> status=live|dark line per type and the workflow keys the issues off those, not off the exit code — exit 2 already means "the probe could not run", and a type the probe never reached must not open an issue on evidence nobody gathered. Build and datastore streams are deliberately out of scope, with reasons recorded in the ADR088 §6 coverage table.
TestShipperRegex* (lego/backend/internal/logs/shipper_attribution_test.go) compiles the regex out of deploy/gitops/base/log-shipper.yaml in CI, so the config and the query cannot drift apart again.

Discovery must not offer what a filter cannot match (w6/m131). host values come from app.Status.URLs rather than the store — the cardinality budget keeps host in the line, not in a label — which is why host discovery resolves even with no store wired. That is kept: it is the correct answer to "which hosts does this service serve", and with attribution fixed the host line filter matches exactly those values. The dashboard's static Method/Status-code/Level fallback lists are narrowed instead: they stand in for "we could not ask" (still loading, or discovery 503s with no store) and are dropped once logLabelValues answers with an empty set, so the Filters popover no longer advertises GET·POST·PUT·PATCH·DELETE on a service whose store says it has produced no request lines.

REST surface

method + path	effect
`GET /v1/logs`	historical query → `{hasMore, next*Time, logs}`
`GET /v1/logs/values`	filter-value discovery → a bare `["…"]` array
`GET /v1/logs/subscribe`	live tail over WebSocket, Server-Sent Events or NDJSON
`graphql { logs(...) }`	same paging envelope, with flat `LogEntry` rows
`graphql { logLabelValues(...) }`	same discovery (the logs sibling of `metricsFilters`)
MCP `list_logs`	agent read (Core.QueryLogs), `resource` array + the full filter set
MCP `list_log_label_values`	agent discovery (Core.LogLabelValues), `label` + `resource` + the same filters

Query params (Render vocabulary): resource (repeatable App, managed-Postgres or Key Value id), plus the filters above — type, level, instance, host, statusCode, method, path, text (all repeatable), startTime/endTime (RFC3339), direction, limit. /v1/logs/values takes the same set plus a required label; Postgres supports instance discovery and durable severity-value discovery. One Core verb per read (QueryLogs, LogLabelValues) backs REST, GraphQL and MCP. An omitted endTime means now. An omitted startTime reads back seven days. That matches retention, so a bare query surfaces stored history; it deliberately differs from Render's documented now() - 1 hour. The page cursors name that same window: they used to default to one hour, which inverted a backward page's next window once its lines were older than an hour, and the next request refused it with a 400 (w8/125). A startTime after now with no endTime therefore names a window that has not begun, and every surface answers it with an empty last page. That includes the Postgres and Key Value log reads. Before w8/120, Loki was sent the inverted range and the read failed with a 500. A deploy's log panel can ask this way, because it starts from the deploy's creation time and the database clock may run a moment ahead of the API's. An explicit inverted range is still a 400. An endTime alone is fine even in the future: the default lookback and the page cursors count back from whichever is earlier, endTime or now (w8/120). The read stops at now (w8/123). No line is stamped after now, so the window sent to the store ends there even when endTime is later. It ends ten minutes past the API's clock, the creation grace Loki allows a node whose clock runs ahead. An endTime inside that grace stays the caller's own inclusive end. Before this, a far-future endTime with no startTime sent the caller's end. Once that lay more than Loki's 720-hour maximum past the default start, the read came back as a 500; just under it, it walked weeks of empty future until a 503 timeout. BEX_MAX_QUERY_HOURS measures the same window that can hold lines, so startTime = one day ago with endTime = 40 days ahead reads that day. A startTime older than the cap is still a 400, and a window that has not begun is an empty page. Metrics differ: an omitted start counts back its default span from endTime, so a far-future endTime alone reads an empty future window, not an error.

Label discovery reads a stream index, not log lines. REST /v1/logs/values, MCP list_log_label_values and GraphQL logLabelValues accept their usual text arguments, but text, path and host do not narrow discovered values. Stream-label filters and time bounds still narrow the Loki index lookup. Preserving every text term in the shared REST/MCP parser does not change that limitation. App host discovery comes from the App's URLs, and datastore instance discovery can fall back to authorized pods; neither source evaluates line text.

Discovery is scoped to the App, always: the label-values call goes to Loki with the requested service's stream selector, so no caller can enumerate another tenant's pods, hostnames or statuses. host is the exception that proves the taxonomy — it is not a stream label, so its values come from the App's own status.urls (core.HostsFromURLs), which is why it resolves even with no store wired. (Logs-only: the metrics feature's discovery deliberately offers no HOST values — its query verb rejects the filter, w3/m12.)

The REST log object is Render's public-API log (all fields required): {id, message, timestamp, labels[]}. bex synthesizes a stable id from instance + timestamp + a message hash, and renders Core's map labels as Render's {name,value} array — type, resource (Core's service), instance, container, level, method, statusCode. A line carries only the labels its stream actually had: an app line has no method/statusCode, a request line no instance/container. Nothing is faked to fill the shape. The envelope carries all four required fields: hasMore, nextStartTime, nextEndTime, logs. The cursors name the next page's window for the query's direction — feed them back as startTime/endTime (Render's documented contract, and the official CLI's scroll-to-load path). Backward (default): nextStartTime = the query's start, nextEndTime = one nanosecond before the page's oldest line (so an inclusive endTime does not re-include the boundary). Forward: nextStartTime = one nanosecond past the page's newest line, nextEndTime = the query's end. Shared-timestamp siblings at a cut are kept on the same page (capToLimit expands the group) so the time cursor cannot drop one. GraphQL logs and MCP list_logs return the same envelope (w4/m107). (MCP's entry objects still use map labels verbatim, matching Render's MCP server — each adapter mirrors its own Render counterpart.)

A search is time-boxed and pages on honestly (w4/m140). A line filter (text, path, host) makes Loki read every line in the window, so a no-match search over a week of a busy service outlasted the 30s HTTP write deadline. The connection was cut before any answer was written, and the edge replied with a CORS-less 502 the dashboard could only report as "Failed to fetch". The durable source now scans a line-filtered window in slices from the end the direction keeps (1h, 2h, 4h, …) and stops when the page fills, the window is covered, or an 18s budget is spent. Any nonempty normalized text set selects this bounded scan, even when earlier supplied terms were empty; multiple terms form one OR predicate within the same budget. A budget-stopped read returns the matches it covered with hasMore:true and the cursor at the coverage edge, so the next page resumes exactly there. All three surfaces share this through one page builder, and across several resources the least-covered one bounds the page. Only when not even the first slice finishes does the read fail, as a named QUERY_TIMEOUT (503 on REST, a coded GraphQL error). Label-only reads stop at limit inside Loki and stay a single request. Separately, GraphQL execution is bounded at WriteTimeout − 5s, and any resolver that runs into it reads QUERY_TIMEOUT rather than a severed connection.

Relative-range reader continuity (w4/m151): the service Logs viewer identifies a reading session by resource, filters and selected range, separately from the moving request bounds. Automatic relative-range ticks retain loaded rows still inside the window and the older-page continuation while refreshing the head; the mounted list preserves an unpinned reader’s visible anchor. A deliberate resource/filter/range change resets the reader and rejects stale completions. Transient refresh errors remain visible alongside retained rows; authorization loss or resource absence clears them. Custom intervals stay fixed. Datastore viewers keep their existing fixed resolved windows, deploy history keeps its deployment-window reset policy, and Metrics/Scaling retain their clocks. The API’s 100-row cap, cursor envelopes and live transport are unchanged. Render’s logging docs, checked 2026-10-02, establish preset/custom/live controls, not this exact idle-clock guarantee; local regression coverage is not a hosted walkthrough or authenticated Render execution.

Selection-time relative clock (w4/m203): selecting a relative Logs or Metrics range samples the clock in that same render, so the first query ends at the selection time rather than at the page's mount or last tick; this covers preset → preset, Custom → preset (the same preset included) and a span-only change. The window then advances at its usual resolution cadence. Custom bounds stay fixed, and the API keeps honoring caller-supplied bounds without resampling.

Logs filters follow navigation (w4/m204): the service Logs viewer treats the URL as the authority for its filters and Live state. A URL it did not write — the service nav link, Back/Forward, a pasted link — is adopted in the same render, so the history query and the live-tail predicate describe the destination at once and the page never shows a narrower reader than its URL names. Its own debounced writes echoing back are recognized and leave newer typing alone. Datastore viewers and deploy log panels keep their separate state models.

One record, one id across history and live (w4/m96): because the id hashes the message, the durable-history reader (parseLokiStreams) canonicalizes each container record's message to the exact bytes the live/fallback pod reader emits — stripping only the single kubelet transport line terminator (\r?\n, precisely bufio.Scanner's ScanLines boundary), never trimming data whitespace, an embedded/JSON-escaped newline, or an empty record. The API-sourced pipelines (loki.source.kubernetes) keep that terminator (Alloy's parseKubernetesLog returns the timestamp-stripped remainder including the LF); the file-sourced ones (type=build, type=app since w4/m174, and type=postgres and type=keyvalue since w5/m122) go through loki.source.file + stage.cri and are already terminator-free; only a final \r, which the live reader drops too, is removed, and type=build is left byte-for-byte untouched. Without this, a line straddling the last historical page and the first live frame carried two message-byte shapes and two ids, so the viewer's key-based merge rendered it twice — a recurrence of the w9/053 duplicate through a different input mismatch. The same reader then applies the live reader's 64 KiB message cap (capLogMessage: a UTF-8-safe prefix plus …[truncated]) to every historical message, all types included, so a record over the cap is returned in one capped shape from history, live, and the pod-log fallback alike rather than whole from history and capped from live (w4/256). Text search, raw and display, matches the returned message, so a term that exists only beyond the cap does not match on any path. Raw timestamps and resume cursors are untouched; a CRI-split record whose historical and live nanosecond stamps differ still derives two wire ids.

Log types

Render's type is app/request/build, and bex serves all three (w7/m28) plus a bex-native predeploy (w1/m33):

app — the App's own container stdout/stderr, from every replica pod (label app.bex.co/app=<name>), aggregated. application is accepted as an input alias.
request — Traefik's access log for that App (see Log labels), with truthful method/statusCode and a searchable path/host.
build — the in-cluster BuildKit/kpack output for a git-backed deploy, shipped by the build_pods pipeline in deploy/gitops/base/log-shipper.yaml. Build pods carry app.bex.co/component=build + app.bex.co/build=<name> (w7/m28); the shipper attributes them to the App and pushes {namespace, app, pod, container, type="build"} streams. Without the durable store (BEX_LOKI_URL unset) a historical type=build query returns 503, not a silent empty — the same honesty rule as type=request. The SSE subscription is independent of Loki and follows the newest running build pod directly (w3/m14). Pre-deploy Job pods (component=predeploy, w5/m100) are tailed by the same pipeline and also land as type=build, so Render-compatible clients whose --type enum is closed over app|request|build can retrieve migration stdout after pre_deploy_failed without the bex-only predeploy type.
predeploy (bex extension, w1/m33) — the pre-deploy step's Job-pod logs (spec.preDeployCommand, ADR004): a live read of the predeploy container on the pod labelled app.bex.co/predeploy=<name> in the App's namespace — the precise bex extension (mixing it with app/request/build is a 400). The same Job's stdout is also shipped into Loki as type=build (w5/m100), so after the pod is TTL-reaped the output remains on the Render-compatible build path; the live predeploy read is still empty once reaped. Instance filters resolve public IDs and legacy raw names against only the authorized App’s extant pre-deploy pods before reading logs (w2/039). Unknown, foreign and App-pod selectors return no lines; text/time filters remain AND constraints. This source does not use historical App-instance discovery or provide a pre-deploy subscription.

Asking for no type (or all) unions app + request — the default a Render client sees. Build logs are only included when the caller explicitly requests type=build; predeploy is likewise a separate, explicitly-requested source, never in that union.

Live tail

GET /v1/logs/subscribe follows every authorized resource: a WebSocket upgrade serves the Render CLI, Accept: text/event-stream selects SSE, and other requests receive NDJSON. Each matching line is emitted once in the selected format; SSE uses data: <log JSON>. type=build selects the active build pod; absent type keeps the established App-pod tail. Text terms OR together on both producers, with the supported time and instance constraints still applying. The handler clears the server write deadline (http.NewResponseController) so the long-lived stream isn't killed by the api's WriteTimeout; the stream ends when the client disconnects (request context cancelled), or with a named terminal error when the requested source cannot be followed. Datastore tails (w5/m139): a dpg-… or red-… resource follows the datastore's pods' process container (CNPG postgres, valkey) with the same per-line parsing as the historical pod read — CNPG instance-manager JSON dropped, a PostgreSQL record unwrapped with its severity as level, a Valkey line's marker as its level — under the same authorization (AuthorizeDatabase/AuthorizeKeyValue, re-checked fresh by the tail watchdog), and instance, text and level narrow it per line. Each resource is validated against its own contract before any streaming header or WebSocket 101, so level is accepted for a datastore and still refused beside an App's pod tail. Pre-deploy tails remain unsupported.

Render compatibility

Shapes verified against render-public-api-1.json and render-oss/render-mcp-server (pkg/logs/tools.go): the full filter param set, the {hasMore, nextStartTime, nextEndTime, logs} envelope, the {id, message, timestamp, labels[]} log object (with the label-name enum), and both MCP tools' names/arguments (list_logs, list_log_label_values — the latter's label enum is Render's exact six). Known, intentional deviations:

subscribe transport — Render-compatible WebSocket (101 upgrade), with SSE and NDJSON alternatives for browser and terminal clients.
ownerId — Render requires it; bex is single-tenant and omits it.
text is literal; structured wildcards are a subset — Render documents wildcard/regex text search; bex ORs case-insensitive literal text terms and supports * in the structured filters named above (see Log filters).
textMode (Bex extension) — Render documents no ANSI-aware search option; textMode=display is bex-only and opt-in, and omitting it keeps Render's raw literal contract (see Log filters).
type=build — requires the durable store (BEX_LOKI_URL); without it returns 503 (w7/m28 — the same honesty rule as type=request).
request-line message — the raw JSON access line, not a rendered request summary.
GraphQL arity — type/text on logs and logLabelValues stay single-valued strings (the shape the dashboard sends for history); generic REST and MCP take Render's arrays. A scalar text value becomes one term without comma splitting. The request filters are lists on all three. The logs field returns Render's {hasMore, nextStartTime, nextEndTime, logs} envelope (w4/m107), not a bare [LogEntry] list.
label discovery — text/path/host line predicates cannot narrow stream-index values, even though the discovery adapters accept these arguments.
get_postgres_logs / get_key_value_logs (MCP) — bex-only convenience tools; they stay a bare {logs:[…]} without the paging envelope. Agents that need to page use list_logs with the dpg-/red- resource id, which carries the same envelope as REST/GraphQL.

RBAC

The api ServiceAccount reads pods (get/list/watch) and pods/log (get) — added with the logs verb in lego/operator/config/api/rbac.yaml. No clientset lives in Core; only podlogs.go (and its main.go wiring) touch it. The Loki-backed history reaches Loki over HTTP (BEX_LOKI_URL), not the kube API, so it needs no extra bex-api RBAC — the log-shipper DaemonSet's own ServiceAccount (its chart's ClusterRole) does the pod-log reads, exactly as the Prometheus scrape's ServiceAccount does for metrics.

Request correlation: `Rndr-Id` and `requestID` (w8/m125)

Render's logging guide tells apps to log the per-request Rndr-Id header and search for it, and its request logs carry the same value as requestID. The bex-client-ip-headers entrypoint plugin (ADR022) mints a random UUID-shaped Rndr-Id for every request. It replaces any caller value, both spellings included, and sets the id on the request to the app and on the response. Traefik's access log keeps that one request header (accessLog.fields.headers.names.Rndr-Id: keep; every other header stays dropped), so each request line carries request_Rndr-Id. bex-api reads it from the line when parsing request entries, as the requestID label. It is never a Loki stream label, since it is unique per request. Request lines shipped before the rollout have none. The dashboard's request row shows the raw access-log line, so the id is visible and searchable there too. Verified on the pinned traefik:v3.7.5: the response header, the app's rndr-id and the access log's request_Rndr-Id were one minted value for a request that sent a forged one.

Service incarnations (w8/m124)

An App's streams, pods and Traefik routers are keyed by its CR name, which follows the service name. Deleting a service and creating another with the same name therefore handed the newcomer its predecessor's history: paths, client IPs, SIGTERM lines and metric points. Render scopes logs and metrics to the service id, so every App read is now scoped to the current incarnation:

Stream label. The log shipper copies the pod's bex.co/app-id into an app_id stream label (deploy/gitops/base/log-shipper.yaml, app_pods). App reads add app_id=~"<id>|" beside the namespace/app tenant matchers. The empty alternative keeps streams shipped before the label, and Job pods, which carry no id label. A predecessor's terminating pod carries its own id and is excluded.
Creation floor. Every log read (history, the initial tail, label values, --tail) and every metric read starts no earlier than the App CR's creationTimestamp. The floor only narrows: an App older than the default 7-day lookback keeps the lookback. It removes the predecessor's unlabeled history and its request lines, which Traefik streams carry no pod label for. Month-to-date bandwidth starts there too.
Pods. Every pod list (live tail, instance discovery and filters, the pod-log fallback, metrics' snapshot paths) drops pods labelled with another id (core.OwnAppPods).
cAdvisor series. Pod names cannot tell incarnations apart, so pod-scoped PromQL ends in unless on(pod) (kube_pod_created{namespace=…} < <creation unix>). That is unless, not and, so a pod kube-state-metrics has no series for still charts.

Postgres and Key Value already select by immutable id.

Metrics

The same one-Core-many-adapters shape as logs. Core.Metrics(MetricQuery) is the single read; REST (internal/metrics/rest.go) and GraphQL (internal/metrics/graphql.go) are the surfaces. Two backends, each an injected source so Core stays clientset-free (like PodLogSource):

flowchart LR
  rest["REST GET /v1/metrics/{cpu,memory,instance-count,<br/>http-requests,http-latency,bandwidth}"] --> core
  gql["GraphQL metrics(...)"] --> core
  core["Core.Metrics"] --> rr["ResourceMetricsRange source"] --> promc["Prometheus ← cAdvisor (kubelets)"]
  core -.fallback.-> rm["ResourceMetrics source"] --> ms["metrics-server (metrics.k8s.io)"]
  core --> pods["App pods (limits + fallback instance count)"]
  core --> qm["RequestMetrics source"] --> prom["Prometheus ← Traefik"]

Resource — cpu / memory / instance_count come from cAdvisor scraped by Prometheus (NewPrometheusResourceSource, gated by BEX_PROM_URL like the request metrics): a query_range over container_cpu_usage_seconds_total (per-pod rate → cores) and container_memory_working_set_bytes (per-pod sum → bytes; also counted for instance_count), one stepped series per replica tagged instance + resource. The public instance label is the name-derived service-instance id (same as serviceInstances / logs), not the raw Kubernetes pod name — PromQL still matches by pod name internally. The cpu and memory sums first take max by (pod, container) so cAdvisor's duplicate series for one container across a restart do not double-count: while a container is OOM-killed and its replacement starts, cAdvisor briefly reports both instances (same pod + container, different cgroup id), and a bare sum by (pod) added them — a Guaranteed 512 MiB App momentarily read ~745 MiB (≈145% of its enforced limit) at every OOM restart, which looked like an enforcement gap when the limit was in fact applied (w4/050, prod-confirmed: single-instance peak 508.9 MiB vs the double-counted 743 MiB; the operator sets requests==limits per tier at Guaranteed QoS, so the pod is OOM-killed at its plan limit rather than exceeding it). instance_count needs no such guard — its inner sum by (pod) already collapses a pod to one series. Since kubelet metrics carry pod names but not pod labels, an App's pods are matched by the Deployment pod-name shape — anchored, so web never matches web-api pods. The selector (egressquery.PodNameRegex, shared with the usage meter) accepts two shapes, because Kubernetes generates only one of them reliably: the untruncated <obj>-<rs-hash>-<5 random>, and the single-segment <obj>-<N alphanumerics> that generateName leaves once the base <obj>-<hash>- passes 58 chars and the cut eats the separating hyphen. A truncated name is always exactly 63 chars, so N is pinned to 62-len(obj) rather than left open — that exact length plus the hyphen-free character class is what keeps a sibling App's pods out. <obj> is the Kubernetes object name core.CRName(tenant, name), never the workspace-scoped public service name (w6/m110). With percentage=true, every sample is divided by its own instance's trustworthy limit at that same timestamp and reported 0..100 — normalization precedes replica aggregation, so mixed-limit replicas (0.4 of 0.5 vs 0.5 of 1 CPU) read 80%/50% and MIN/MAX/AVG aggregate to 50%/80%/65% (w5/m90). The denominator is the limit history (NewPrometheusResourceLimitSource: query_range over kube-state-metrics' kube_pod_container_resource_limits, summed by pod, same Prometheus as the usage history — no new scrape config), joined per timestamp with step interpolation, so a rollout mid-window keeps the old limit for old samples. Samples with no trustworthy denominator — a deleted pod (never another pod's limit), a sample predating limit retention, a zero limit — are omitted, never zero-filled or borrowed; a limit-source failure rejects the read rather than half-joining. Without a limit source (a test/dev shape — production wires both together) the pods' current spec limits stand in. Every tiered App has one (see ADR003-control-plane.md's tier catalog); this path only fires for a bare-CR App with no spec.tier set.
Resource fallback — without BEX_PROM_URL, cpu / memory come from metrics-server (metrics.k8s.io/v1beta1, via NewResourceMetricsSource): a point-in-time snapshot, so each series carries a single current point regardless of the requested range. instance_count then derives from the App's pods directly, needing no source at all. When Prometheus is configured but unreachable at query time, the error surfaces (no silent fallback — same contract as request metrics).
Request — http_requests / http_latency come from Traefik scraped by Prometheus (NewPrometheusRequestSource, gated by BEX_PROM_URL): a query_range over traefik_service_requests_total (rate) and _request_duration_seconds_bucket (histogram_quantile, default p95). bandwidth is the App's complete outbound rate: exact Traefik router response bytes + WebSocket downstream frames + direct public L3 bytes, using the same egressquery vocabulary as hourly usage. Since w1/m50 the interactive read is best-effort: a source failing its health product no longer errors the window — the series is served with a degraded_sources label (and monthToDateBandwidth a degradedSources field) naming the unhealthy sources, while the hourly usage rollup keeps the strict billing gate (ADR023 § Observability reads vs billing reads). statusCode filters the HTTP code label (2xx → 2..); groupBy (status/method) applies to HTTP request/latency series.

When a metric's source isn't wired at all (request metrics without BEX_PROM_URL, cpu/memory with neither Prometheus nor metrics-server), the endpoint returns 503 (ErrMetricsUnavailable) — the App exists, the data source doesn't.

Everything above is the TENANT-facing read: metrics about a tenant's App. bex-api's telemetry about itself — the origin-side per-route request histogram, the GraphQL-operation and MCP-tool families, and the two BexApiOrigin* alert rules — is platform-side, exposed on the internal :8091 /metrics registry and documented in ADR088 §6. It never reaches a tenant surface, and it obeys the same cardinality rule as everything else here: route labels are registered mux patterns, never paths, and tool/operation labels come from closed registries. Nothing in this section changed when it landed (w3/m84).

REST surface

method + path	metric
`GET /v1/metrics/cpu`	per-instance CPU (cores, or % of limit)
`GET /v1/metrics/memory`	per-instance memory (bytes, or % of limit)
`GET /v1/metrics/instance-count`	running replica count
`GET /v1/metrics/http-requests`	request rate (req/s)
`GET /v1/metrics/http-latency`	latency percentile (seconds)
`GET /v1/metrics/bandwidth`	outbound bytes/s

Query params (Render vocabulary): resource (App id, repeatable; service is accepted as its alias), startTime/endTime (RFC3339), resolutionSeconds, quantile (0..1, latency), statusCode (request filter), host/path (request filters served from the log store, w5/m58), aggregateBy=statusCode|host on http-requests (one series per status code, labelled statusCode, or per request host, labelled host — w1/m155 retired the non-Render groupBy; w5/m140 served host from the request-log store, which never falls back to the host-less Traefik counter, so with no store it is the host filter's 503), aggregationMethod=AVG|MAX|MIN on cpu (Render's interval aggregation, w5/m140: MAX/MIN report each step's highest/lowest per-instance rate via a max_over_time/min_over_time subquery at the 15 s scrape grain; AVG/omitted is the plain rate; a named 400 on any other metric, or when cpu comes from the metrics-server snapshot), bex extras percentage=true (cpu/memory as a fraction of limit), instance (repeatable public instance ids from w5/m87; omit for all), and aggregateAllMethod=MIN|MAX|AVG (replica aggregate at each timestamp — w5/m89; distinct from the interval aggregationMethod, which MCP spells cpuUsageAggregationMethod and Render's MCP server passes straight to REST). Eligibility is metric-typed, not data-derived (w5/m91): instance is honored on cpu/memory/cpu_limit/memory_limit (the per-instance series, including the m89 limit consumers), so an authorized empty window succeeds with no series instead of misreporting a bad request; on any other metric it is a named 400 even when empty. Unresolved or foreign instance values yield an empty series rather than silently selecting everything. The dashboard retains an explicit selection across discovery gaps and window changes until an explicit Show-all or a resource navigation clears it, marking retained-but-unavailable choices instead of pruning them. Each endpoint returns Render's metrics array — [{labels:[{field,value}], unit, values:[{timestamp,value}]}]. GraphQL mirrors Render's dashboard shape: metrics(query: MetricsQueryInput!) — input fields filters (resource selectors, including INSTANCE), name (the metric: cpu/memory/instance_count/http_requests/http_latency/bandwidth), start/end, resolution, parameters, aggregateBy, aggregationMethod, aggregateAllMethod, plus the bex-extension percentage boolean (w5/m90 — no Render equivalent; mirrors REST ?percentage=true and MCP percentage) — returning MetricSeries { unit, labels{field,value}, values{time,value}, parameters } (the sample field is time in GraphQL, timestamp in REST). Companion dashboard queries: monthToDateBandwidth, metricsFilters, metricsPathFilterSuggestions. The MCP get_metrics tool (resource[] + metricTypes[]) exposes the same read to agents — three-adapter parity, like list_logs.

Render compatibility

Shapes track Render's metrics endpoints (per-metric path segments; the {labels, unit, values} time-series). With Prometheus configured (BEX_PROM_URL), all six metrics are resolution-stepped series honoring startTime/endTime/resolutionSeconds — Render metrics-page parity. Known, intentional deviations:

snapshot fallback — without BEX_PROM_URL, resource metrics fall back to metrics-server and return a single current point (metrics-server has no history); Render always returns a stepped series.
cpu_limit/memory_limit stay single-point — limits come from the current pod spec, and bex won't fabricate a history for a value it only knows now (a past limit may have differed). Stepped limit history for percentage denominators comes from the separate kube-state-metrics source above, not from these endpoints. The dashboard no longer divides client-side (w5/m90): it reads server percentage series directly, and these single-point series feed only the header label and the Total tab's reference line — a uniform limit reads as one value, mixed per-replica limits read as "Limits vary", never one max presented as universal.
host/path filters — rejected with a 400 naming them (w3/m12; confirmed infeasible w3/m18): Traefik's Prometheus counters (service- and router-level) intentionally carry no host or path labels — adding them would be unbounded cardinality. addRoutersLabels: true is already enabled in the Traefik config and adds a router label (the router name, e.g. my-app@kubernetes), not the matched Host() or PathPrefix() values from the routing rule, so router-level metrics are equally unlabelled by host/path. Host/path-scoped request analysis requires parsing the access log (type=request in Loki, the logs API with path/host filters). A query with a host or path filter cannot be answered honestly and is refused rather than silently answered with whole-service series. GraphQL errors on a HOST/PATH filter entry identically; MCP's get_metrics doesn't expose the parameters at all; and the metricsFilters discovery verb reports empty HOST/PATH values, so no client is offered a filter value the query verb refuses.
Traefik service selector — the App→Traefik-service match for HTTP count/latency is a heuristic (service=~".*<app>.*"); bandwidth instead uses exact operator-owned router labels. The resource-metrics pod match (pod=~"<obj>-[a-z0-9]+-[a-z0-9]{5}|<obj>-[a-z0-9]{62-len(obj)}", egressquery.PodNameRegex) is the stricter cAdvisor sibling. Its two-segment-only predecessor returned no series at all for any App whose object name crossed generateName's 58-char truncation point — a service name of ~22 characters, well inside the 30 ValidResourceName allows — and the same matcher in the usage meter, called with the workspace-scoped store name instead of the object name, metered every App's compute as a healthy zero for the meter's whole life (w6/m110).

RBAC

The metrics-server fallback adds read on metrics.k8s.io pods (get/list) to the api ServiceAccount (lego/operator/config/api/rbac.yaml); percentage mode reuses the existing pods read for limits. The Prometheus-backed metrics (resource history and request) reach Prometheus over HTTP (BEX_PROM_URL), not the kube API, so they need no extra RBAC on bex-api — the Prometheus ServiceAccount's chart-default ClusterRole covers the cAdvisor scrape (nodes, nodes/proxy, nodes/metrics).

Cluster enablement

deploy/gitops/base/prometheus.yaml runs the one Prometheus behind both history-backed metric families. Two scrape jobs feed bex-api's metrics: traefik (request counters, via deploy/gitops/base/traefik.yaml's metrics entrypoint :9100 with addServicesLabels) and kubernetes-cadvisor (per-container cpu/memory, scraped through the apiserver proxy so it works even where the pod network can't reach every kubelet). Four more feed the alerting rule pack below — kube-state-metrics (object state), kubernetes-kubelet (only kubelet_volume_stats_* for PVC usage), cert-manager (:9402 certificate series), and openbao (per-pod /v1/sys/metrics for seal state). A fifth, cnpg-tenant-db (w3/m10), scrapes every managed-Postgres CNPG pod's :9187 exporter across all app namespaces (bex's own bex-db stays on its own tightly-scoped cnpg-bex-db job), keeping cnpg_backends_total + cnpg_pg_replication_* — the extended-metrics series below. BEX_PROM_URL points bex-api at the server and enables the metric families. deploy/gitops/base/metrics-server.yaml installs metrics-server — now only the snapshot fallback (and kubectl top).

Production replay budget (w4/m144, 2026-10-02)

Production Prometheus exhausted its 512 MiB container limit while replaying the existing WAL. At 08:14:24 UTC the last exit was OOMKilled (137), with 577 restarts and no ready endpoint. The 10 GiB provisioned PVC still held about 1.48 GiB; retention alone does not bound replay memory. Prometheus keeps current head data in memory and reconstructs it from the WAL on startup (storage documentation).

The production overlay now requests 1 GiB and caps the server at 2 GiB; the shared/local budget stays 256/512 MiB, and CPU stays 50/500m. Both overlays use Helm parameters so they preserve the shared inline values. The local overlay previously used valuesObject, which Argo selects instead of inline values; the validation helper now models that choice correctly, and the local parameters restore the intended shared rules, storage and budget. This is a bounded recovery budget, not a measured successful replay peak. The hosting node had 7,651 MiB allocatable, 5,526 MiB of pod requests and about 3,599 MiB available at 08:18 UTC. Raising the request adds 768 MiB, leaving about 1,357 MiB of schedulable memory there. A full 2 GiB server working set would leave approximately 1.5 GiB of the observed available memory; that is a point-in-time capacity check, not a guarantee against concurrent workload peaks.

Keep the existing PVC, WAL, three-day retention, scrape jobs and rules. Before closing the incident, the platform release/QA owner must observe replay completion on that dataset, measure replay and query/steady-state peaks plus head-series cardinality, recheck node capacity, and verify an existing-data restart with a stable restart count for 15 minutes. Confirm fresh scoped REST/GraphQL/MCP series and the dashboard charts, alongside the pod-spec limit controls. Preserve any historical gaps and distinguish unavailable sources from empty windows; this recovery does not establish billing backfill. The detailed consumer matrix and remaining gate are in w4/m144 verification.

Extended metrics: autoscale-target, disk, DB connections/replication-lag (w3/m10)

Four bex-extension series closing the render-parity ledger's last open metrics row (docs/ADR018-render-parity.md's "Extended metrics"). The first is App-scoped (Core.Metrics, same verb as cpu/memory); the other three are Database/KeyValue-scoped — a new Core.DatastoreMetrics verb (internal/metrics/datastore.go), since the resource isn't an App and can't go through s.GetApp. It re-resolves the Database/KeyValue CR by name (AuthorizeLabeled, the same cross-workspace gate internal/postgres/internal/keyvalue's own fetch helpers apply) rather than importing those feature packages — features never import each other.

cpu_target/memory_target — the App's configured autoscale-target utilization percentage (spec.autoscaling.targetCPUPercent/targetMemoryPercent, w1/m20), a single current-value point like cpu_limit/memory_limit (a config value, not a usage sample). Omitted — not a fake zero — when autoscaling is disabled or the specific target isn't set.
disk/disk_capacity — a managed Postgres or Key Value instance's backing-PVC used/capacity bytes, via query_range over kubelet's already-scraped kubelet_volume_stats_{used,capacity}_bytes{namespace,persistentvolumeclaim=~pattern} (no new scrape config — see Cluster enablement above). The PVC name pattern is derived from the operator's own naming: <name>-\d+ for a Database (CNPG's per-instance PVCs), data-<name>-\d+ for a KeyValue (its StatefulSet's data volumeClaimTemplate).
db_connections — a managed Postgres instance's live active-connection count, via query_range over CNPG's postgres_exporter cnpg_backends_total (its backends default-monitoring query's total column, summed across every datname/usename/state), scoped by the cnpg-tenant-db scrape job's cnpg_io_cluster label. Postgres-only — DatastoreMetrics errors if asked for a KeyValue resource.
replication_lag — a managed Postgres instance's replication lag in seconds, via CNPG's cnpg_pg_replication_lag (its pg_replication default-monitoring query's lag column). Gated, not degraded: the verb never queries Prometheus for this metric unless Database.status.highAvailabilityEnabled is true — without a standby CNPG's own query returns 0 from a lone primary (not absence), which is exactly the fake-zero the omit-don't-fake rule (above) exists to avoid. An HA Postgres (w1/m22) returns a real series; non-HA Postgres returns nil and the dashboard renders a clear N/A state rather than a broken chart. (w3/m17)

REST: GET /v1/metrics/{cpu,memory}-target?resource=<app> (same shape as the other App-scoped endpoints) and GET /v1/metrics/{disk,disk-capacity,db-connections,replication-lag}?resource=<name>&kind=database|keyvalue (kind defaults to database). GraphQL: CPU_TARGET/MEMORY_TARGET join metrics' name enum; the other three ride a new datastoreMetrics(query: DatastoreMetricsQueryInput!) query — {kind, resource, name, start, end, resolution}, naming one resource directly rather than a RESOURCE filter array (a datastore metric always targets exactly one instance). MCP: cpu_target/memory_target join get_metrics' metricTypes; the datastore trio gets its own get_datastore_metrics tool (resource, kind, metricTypes[]). All four are bex extensions with no literal Render endpoint — Render's dashboard shows autoscale-target/disk/connection info as part of other views, not this metrics API shape — so parity here means "the same data, reachable the same way as bex's other metrics," not a captured Render wire format.

Platform alerting (Alertmanager)

Logs and metrics above make tenant deploys observable. Platform alerting is the operator-facing half of GOAL.md #2: when bex itself breaks — a bad rollout in bex-system, a node gone, OpenBao sealed after a restart, a nightly backup silently rotting — a human gets paged instead of finding out at restore time. It rides the same Prometheus (w3/m6): the chart's bundled Alertmanager is enabled with an email receiver, and a small, high-signal rule pack (deploy/gitops/base/rules/alerting_rules.yml, mounted at /etc/rules from ConfigMap monitoring/prometheus-alerting-rules and watched by the configmap-reload sidecar; it moved out of the inline prometheus.yaml Helm values in w3/m177 because Argo retains those values roughly eightfold and the Application ran out of its size budget) evaluates platform and bex-specific invariants.

Collection stays scoped to the rules it serves: kube-state-metrics provides object state, the kubelet provides PVC usage, and a filesystem-only node_exporter provides tenant-node ephemeral-storage headroom. No pushgateway or unrelated host collector suite is introduced.

The rule pack

The groups carry actionable descriptions and links to diagnosis procedures:

group	alert	fires when	severity
`platform`	`PlatformPodCrashLooping`	a container in a platform namespace is CrashLoopBackOff >10m	warning
`platform`	`PlatformDeploymentNotReady`	a platform Deployment has < desired available replicas >10m	warning
`platform`	`PlatformGitOpsOutOfSync`	an expected Argo Application reports a non-Synced state for 15m	warning
`platform`	`PlatformGitOpsUnhealthy`	an expected Argo Application reports a non-Healthy state for 15m	warning
`platform`	`PlatformGitOpsApplicationMissing`	an expected Application lacks complete sync/health telemetry for 10m while the Argo scrape succeeds	warning
`platform`	`PlatformGitOpsMetricsMissing`	the Argo application scrape is absent or has no successful target for 5m	warning
`platform`	`ControlPlaneNodeNotReady`	a control-plane node is NotReady >5m (single CP node = high blast)	critical
`platform`	`NodeNotReady`	a worker node is NotReady >5m	warning
`platform`	`NodeDiskPressure`	a node's kubelet reports DiskPressure >2m (an eviction threshold has been crossed)	critical
`platform`	`NodeDiskPressureRecovered`	a node had DiskPressure within the last 6h and has recovered	info
`platform`	`NodeDiskPressureSignalMissing`	a node (or every node) has no kube-state-metrics DiskPressure series >15m	warning
`platform`	`TenantNodeFilesystemSpaceLow`	an expected tenant backing device has <20% available bytes for 10m	warning
`platform`	`TenantNodeFilesystemInodesLow`	an expected tenant backing device has <10% free inodes for 10m	warning
`platform`	`TenantNodeFilesystemTelemetryMissing`	an expected tenant node/mount lacks complete usable filesystem evidence for 10m	warning
`platform`	`TenantNodeFilesystemInventoryMissing`	kube-state-metrics exports no node pool labels for 10m	warning
`platform`	`TenantNodeFilesystemFillingSoon`	an eligible falling trend projects <15% available bytes within 4h for 15m; advisory	warning
`platform`	`EtcdMetricsMissing`	a control-plane node has no successful `etcd` scrape (or none at all) >10m — monitoring loss, not an outage	warning
`platform`	`EtcdNoLeader`	an etcd member reports no Raft leader >1m	critical
`platform`	`EtcdFrequentLeaderChanges`	>3 leader changes in an hour on a member	warning
`platform`	`EtcdHighFsyncLatency`	WAL fsync p99 >500ms for 10m (etcd-mixin threshold)	warning
`platform`	`EtcdHighCommitLatency`	backend commit p99 >250ms for 10m (etcd-mixin threshold)	warning
`platform`	`EtcdDatabaseNearQuota`	DB size >80% of the backend quota for 10m	warning
`platform`	`PersistentVolumeFillingUp`	a PVC is >85% full >15m (hcloud-csi/local-path single-copy volumes)	warning
`platform`	`CertificateNotReady`	a platform cert-manager Certificate is not-Ready >15m — every not-Ready cert except a tenant's own custom-domain cert (which bex can't fix; see the next row)	warning
`platform`	`TenantCustomDomainCertNotReady`	a tenant custom-domain cert (`tea-` ns, `<app>-tls-<host>` for a non-`onbex.co` host) is not-Ready >1h — almost always customer DNS not pointing at bex (e.g. a Cloudflare-proxied apex ⇒ ACME HTTP-01 404s). Routed to the `null` receiver (dashboard-surfaced, never paged); the platform `<app>-tls` onbex host cert stays on `CertificateNotReady` because bex owns `.onbex.co` DNS	info
`platform`	`CertificateExpiringSoon`	an issued (Ready) Certificate expires in <14d and hasn't renewed — joined on `ready_status == 1` because cert-manager exports expiry `0` for a never-issued cert (which made "0 − now < 14d" page for weeks), with the same tenant custom-domain carve-out as `CertificateNotReady`	warning
`platform`	`TenantCustomDomainCertExpiringSoon`	a tenant custom-domain cert was issued but is not renewing (<14d left) — the domain's DNS moved after issuance; the info-tier twin of the row above, `null`-routed	info
`bex`	`BackupCronJobStale`	`etcd-backup`/`openbao-backup` last succeeded >26h ago (silent rot)	critical
`bex`	`PlatformDatabaseBackupStale`	a platform CNPG primary's reported last archived WAL timestamp is older than 26h	critical
`bex`	`PlatformDatabaseBackupTargetMissing`	an expected platform database has no successful primary scrape for 10m	warning
`bex`	`PlatformDatabaseBackupTelemetryMissing`	an expected platform database has a successful primary scrape but no current archiver timestamp for 10m	warning
`bex`	`ZotGarbageCollectionFailure`	the known missing-repository-metadata GC failure was freshly observed within the last 3h	warning
`bex`	`ZotGarbageCollectionTelemetryMissing`	fresh Zot log evidence from a successfully scraped Alloy instance has been unavailable for 10m	warning
`log-delivery`	`LogDeliveryEntriesDropped`	final Alloy writer drops were observed within 15m	warning
`log-delivery`	`LogDeliveryLokiRejected`	Loki ingestion rejection attempts were observed within 15m	warning
`log-delivery`	`LogDeliveryRetriesPersistent`	retry activity appears in every rolling 5m window for 10m	warning
`log-delivery`	`LogDeliveryCollectorMissing`	an expected node lacks a successful Alloy scrape for 10m	warning
`log-delivery`	`LogDeliveryInventoryMissing`	the independent node inventory is absent for 10m	warning
`log-delivery`	`LogDeliveryCollectorUnhealthy`	a successfully scraped expected Alloy collector has faulty or incomplete configuration/component evidence for 5m	warning
`log-delivery`	`LogDeliveryLokiMetricsMissing`	complete Loki scrape/process/WAL-counter evidence is unavailable for 5m	warning
`log-delivery`	`LogDeliveryLokiWALDiskFull`	a Loki WAL disk-full write failure was observed within 15m	warning
`bex`	`OpenBaoSealed`	any OpenBao member reports sealed >5m (⇒ 503s the env-vars API)	critical
`bex`	`BexApiDown`	`bex-api` has zero available replicas >5m	critical
`bex`	`BexApiOriginHighErrorRate`	>5% of the requests bex-api itself completed on one surface are 5xx for >10m (above a 0.1 req/s floor) — the origin-side sibling of `TraefikHigh5xxRate`, read together with it to place the fault (ADR088 §6)	warning
`bex`	`BexApiOriginLatencyHigh`	bex-api's own p95 on one surface is above the recorded 2.5s baseline for >10m (above the same floor)	warning
`bex`	`WebhookDeliveryAdmissionPressure`	>100 outbound webhook notifications remain capped in rolling 15m windows for >10m	warning
`bex`	`WebhookDeliveryFailing`	>90% of outbound webhook attempts fail for >30m, above a 36-attempts/30m floor	warning
`bex`	`PushDeliveryStale`	push is configured, rows are queued, and no provider operation has succeeded in >2h (for >30m)	warning
`bex`	`AgentSessionProvisionFailing`	>25% of agent-session sandbox provisions fail over an hour, above a 10-provisions/hour floor	warning
`bex`	`ClusterBuilderNotReady`	kpack `ClusterBuilder` `bex` missing/unknown/not Ready >15m	warning
`bex`	`ClusterBuilderImageStale`	committed builder resolution missing, malformed, or older than 30 days	warning
`bex`	`StrandedNodeLocalImages`	an App pod is ImagePullBackOff/ErrImageNeverPull >10m	warning
`bex`	`TraefikHigh5xxRate`	>5% of edge requests are 5xx for >10m (above a traffic floor)	warning

etcd members (w7/m160). kubeadm binds etcd's metrics listener to 127.0.0.1:2381, and the client port needs an etcd client certificate — full keyspace access — so neither is scraped directly. deploy/gitops/charts/etcd-metrics runs one kube-rbac-proxy per control-plane node (host network, bound to the node's InternalIP only — tenant pods cannot reach node IPs) that forwards to the loopback listener. Each scrape is authenticated (TokenReview) and authorized for GET /metrics (SubjectAccessReview — the grant prometheus-server already has); TLS is a cert-manager private CA whose serving cert names etcd-metrics.monitoring.svc, which the etcd job pins with server_name so node-IP targets still verify. Prometheus mounts only the CA's public ca.crt. The job keeps a finite allowlist (leader, proposals, slow applies/reads, heartbeat failures, WAL fsync / backend commit / peer RTT histograms, DB size and quota). Scrape loss (EtcdMetricsMissing, per node) is deliberately separate from health (EtcdNoLeader, latency): one unreachable proxy is never read as a quorum problem. The slow-apply / ReadIndex / fdatasync log lines that motivated this are evidence for visibility, not proof that disks need replacing — compare the latency panels with the thresholds before acting.

Node disk pressure (w7/m159). The signal is kube-state-metrics' kube_node_status_condition{condition="DiskPressure"} — the kubelet's own eviction-threshold condition for node-local ephemeral storage (container writable layers, emptyDir, logs, images). PVCs are a separate budget (PersistentVolumeFillingUp). NodeDiskPressure reports an eviction threshold already crossed; the kubelet attempts resource reclaim and may evict pods, which is why it pages. The tenant filesystem headroom signals add a separate warning margin and advisory forecast; they do not replace this current-pressure condition. A shorter episode (the 2026-09-28 03:38 UTC EvictionThresholdMet on bex-tenant-0-bdn2q-bnzl6 cleared by 03:43) surfaces as the info-only NodeDiskPressureRecovered for 6h, and Prometheus keeps the condition series for its 3-day retention; Kubernetes Events expire after an hour, so read them first. Diagnose without deleting: kubectl describe node <node> (conditions, allocatable ephemeral storage), kubectl get events -A --field-selector involvedObject.name=<node>, then the pods on the node — kubectl get pods -A --field-selector spec.nodeName=<node> — looking for Evicted pods and their ephemeral-storage limits (a pod exceeding its own limit is evicted by the kubelet without node DiskPressure; that is the tenant's limit, not node capacity). Resizing machines or pruning images is a separate, evidence-backed decision.

ControlPlaneNodeNotReady vs NodeNotReady split on kube_node_role{role="control-plane"}: the CP pool is a single node until the quota lift restores 3 CP nodes, so its loss pages while a worker's only warns. OpenBaoSealed reads the per-pod telemetry gauge vault_core_unsealed (from the openbao scrape) — not readiness or the Service: the chart's readiness probe keeps a sealed member in rotation (sealedcode 2xx) so the round-robin Service and a kube_statefulset_ready check would both miss a sealed follower; a sealed member still serves /v1/sys/metrics reporting 0, and the alert fires on any member sealed. StrandedNodeLocalImages catches the node-local-image failure mode (App images are ctr imports, not registry-backed, so node replacement/scale-down strands them) — a platform defect, hence warn not page, even though it fires in the tenant default namespace. BackupCronJobStale reads kube_cronjob_status_last_successful_time from kube-state-metrics (the local overlay removes both backup CronJobs, so the series — and this alert — exist only where the jobs run: prod). ClusterBuilderNotReady / ClusterBuilderImageStale (docs/ADR060 D7) read operator-exported unlabeled gauges: readiness is the live kpack condition, age is the committed resolved_at in toolchain-freshness.json, never a mutable tag. Digest movement opens .github/workflows/build-toolchain-freshness.yml's tracking issue; accepting a digest remains a reviewed commit.

Tenant-node filesystem headroom

Production topology verified 2026-09-29 05:45 UTC. The four bex.co/pool=tenant nodes (bex-tenant-0-bdn2q-{2252c,5jjf9,bnzl6} and bex-tenant-lg-4wvlk-s5z8z) share one backing device per node: /dev/sda1, ext4, mounted at /, serving /var/lib/kubelet, /var/lib/containerd and /var/log/pods. Available bytes were 44.53–79.91% and free inodes 75.05–92.30%; inspection was read-only and did not induce pressure. The kubelet configuration sets hard nodefs.available<10%, imagefs.available<15%, and both inode thresholds at 5%, with no soft thresholds or minimum reclaim. Image GC high/low watermarks are 85%/80%; the 5m pressure transition period controls condition oscillation, not a hard-eviction grace period. CAPD's local eviction thresholds are zero, so these warning margins describe the production shape.

Deployed proof, 2026-09-29 06:09–06:12 UTC: feature cc410dc53 plus history-size repair caef683d2 are live. All four intended exporters are Ready and UP; each exposes /dev/sda1 at /, and expected-minus-observed coverage is empty. The live configuration and Cluster capacity panels 18–24 match the shipped render. All five alerts and ten records evaluate healthily, with no filesystem alert active. Each scrape retains nine samples; observed scrape duration was 4.4–7.2ms and filesystem collection 1.1–2.5ms. Each pod used about 1m CPU and 4–5Mi memory. These are point-in-time observations, not a load benchmark. Available-byte ratios were 33.8–78.9% and free-inode ratios 69.7–91.5%. All four forecasts correctly report insufficient history until their two-hour window matures; pressure and recovery were exercised only in isolated fixtures.

The existing kubelet scrape keeps PVC statistics and cannot provide these host filesystem families. The chart's filesystem-only node_exporter supplies available/total bytes and free/total inodes on tenant nodes through the private tenant-node-filesystem scrape. Independent bex:tenant_node_filesystem_expected{node,mountpoint} rows come from kube-state-metrics' tenant pool labels, with / required for the verified topology. Complete evidence requires a successful scrape and all required raw families on a writable, error-free mount. Records aggregate by node/device, so aliases of the same device do not duplicate warnings. A future separate kubelet/image/log filesystem requires updating the expected mount inventory and collector selection together; this root inventory does not claim coverage of undeclared storage.

The locked chart's node_exporter v1.8.2 enables only the filesystem collector. Its host /, /proc and /sys mounts are read-only; it uses no host network, PID/IPC namespace, host port or ServiceAccount token. Each tenant-node pod requests 10m CPU/32Mi memory and is limited to 100m/64Mi; the private scrape runs every 60s. Collector exclusions avoid pseudo/pod mounts before statfs, and ingestion retains only /, /var, /var/lib, /var/lib/containerd and /var/lib/kubelet candidates. Only independently required mounts feed alerts. The local override permits CAPD's overlay root; production excludes overlay filesystems.

TenantNodeFilesystemSpaceLow warns below 20% available for 10m, before the shared device's stricter 15% imagefs eviction boundary. TenantNodeFilesystemInodesLow warns below 10% free for 10m, before the 5% boundary. These are chosen operational margins, not a measured minimum lead time. Missing expected node/mount evidence warns separately after 10m, allowing shorter exporter startup and scheduling gaps; inventory disappears when a node leaves the declared tenant pool. Loss of all node pool labels has its own inventory warning, so an empty expected set cannot silently establish health. A functioning Prometheus evaluator remains necessary.

The advisory fill forecast requires at least 110 samples over 2h, a sample in the 5m window at the start of that history, and unchanged capacity across the window. With that history, it requires available space below 35%, negative 30m and 2h slopes, and a 2h linear projection below 15% available in 4h, persisting for 15m. The panel distinguishes insufficient/unstable history from an eligible forecast with no current risk, and missing telemetry from both. Garbage collection or capacity changes can reset or invalidate that evidence; forecast inactivity does not override a low-headroom warning.

Read Cluster capacity in four parts: the existing DiskPressure condition, current available bytes/inodes, collection coverage, and the advisory forecast. Node-local ephemeral/image storage, PVC capacity, and a pod's ephemeral-storage limit are separate budgets. A pod can exceed its own limit while the node has ample space; conversely, many individually compliant pods can exhaust the shared backing filesystem. DiskPressure indicates a kubelet threshold has been crossed; it does not prove that a particular pod has already been evicted. See the Kubernetes node-pressure model.

Start with read-only evidence for the node named in the alert:

storage_node="NODE_FROM_ALERT"
kubectl describe node "$storage_node"
kubectl get events -A --field-selector "involvedObject.kind=Node,involvedObject.name=$storage_node" --sort-by=.lastTimestamp
kubectl get pods -A --field-selector "spec.nodeName=$storage_node" -o wide
kubectl get --raw "/api/v1/nodes/$storage_node/proxy/configz" | \
  jq '.kubeletconfig | {evictionHard, evictionSoft, evictionMinimumReclaim, evictionPressureTransitionPeriod, imageGCHighThresholdPercent, imageGCLowThresholdPercent}'
kubectl get --raw "/api/v1/nodes/$storage_node/proxy/stats/summary" | \
  jq '{nodefs: .node.fs, imagefs: .node.runtime.imageFs, containerfs: .node.runtime.containerFs}'

On the host, use findmnt -T on the kubelet/runtime storage paths to confirm which actual mount backs them, then compare df -B1 and df -i on that mount with the exporter. Inspect pod eviction messages and limits, runtime image usage, and recent write/log growth before choosing a response. Missing telemetry calls for checking node inventory, exporter scheduling, scrape health and mount selection; another node's healthy series cannot establish this node's coverage. These alerts do not delete images or data, expand disks, replace nodes or change pod limits automatically.

Forecasts are advisory extrapolations of past samples. Sudden writes, image pulls, workload placement changes and garbage collection can invalidate the trend; neither an inactive forecast nor a green historical chart guarantees advance notice. Confirm present byte/inode headroom and actual kubelet thresholds before acting, and keep warning delivery time separate from the rule's persistence window.

Platform GitOps delivery

Prometheus retains three Argo Application revision-history entries because its inline rule pack is copied into each entry. On 2026-09-29, the default ten entries grew the Application to 1,567,464 bytes and etcd rejected the m164 update. A resourceVersion-guarded trim retained the newest three entries after saving the prior object; Git retains older configurations. The declarative revisionHistoryLimit: 3 prevents that accumulation, and rendered validation budgets the repeated source data below etcd's request ceiling. This limits Argo's local rollback list, not Prometheus metric retention. See Argo Application history.

The private argocd-applications job scrapes argocd-metrics.argocd.svc:8082 every 60s. It retains only argocd_app_info and its namespace, name, sync_status, health_status labels plus scrape identity (job, instance); revision SHAs, repository URLs and arbitrary Application labels are excluded. No public endpoint or administrator credential is introduced.

bex:platform_gitops_expected{namespace="argocd",name="…"}=1 is an independent recording rule in serverFiles.platform_gitops_expected.yml, derived from constants rather than exporter output. The base inventory names the production root bex-platform-prod and its children; the local overlay replaces that inventory with its own root and children. scripts/gitops-validate.sh compares each rendered inventory with the corresponding bootstrap root and child Applications, so adding or removing an Application requires updating its expectation. Loss of the exporter cannot erase the expected population.

PlatformGitOpsOutOfSync and PlatformGitOpsUnhealthy retain only the stable namespace/name identity and warn after 15m; changing among failing state labels does not restart their timers. The 15m grace is a chosen maintenance allowance for normal reconciliation, not a measured rollout or notification guarantee. Missing Application state gets 10m, and missing/failed scrapes get 5m. Application-missing alerts require a successful scrape so a controller metrics outage has one diagnosis rather than one alert per missing child. Recovery clears the corresponding condition; email follows the warning digest's separate timers below.

The Platform availability dashboard has a current expected-Application table, per-Application sync and health history, and scrape health. The table preserves native state names such as OutOfSync, Degraded and Progressing; its PRESENT value means telemetry exists, not that the Application is healthy. Missing/incomplete native data or a failed scrape produces UNKNOWN (-1) from the independent inventory, never a green zero. All native panel operands require a successful scrape, so a cached healthy sample cannot mask target loss.

Diagnosis is read-only first:

# Confirm root and child state, then inspect the operation that stopped delivery.
kubectl -n argocd get applications.argoproj.io
kubectl -n argocd get applications.argoproj.io bex-platform-prod -o yaml
application=prometheus # replace with the Application named in the alert
kubectl -n argocd get applications.argoproj.io "$application" -o yaml
kubectl -n argocd get events --sort-by=.lastTimestamp

# Separate target failure from missing Application telemetry.
kubectl -n argocd get service argocd-metrics
kubectl -n argocd get endpointslices -l kubernetes.io/service-name=argocd-metrics
kubectl -n argocd logs statefulset/argocd-application-controller --since=30m
kubectl -n monitoring port-forward service/prometheus-server 9090:80

In the root and child YAML, inspect status.conditions, status.operationState.phase/message, and status.operationState.syncResult.resources for the failed resource and its message. Check that resource's argocd.argoproj.io/sync-wave annotation and compare earlier-wave resources with the child's desired state: a root failure in an early wave can withhold updated child Application specifications even while those children still report Synced. In the port-forwarded Prometheus UI, inspect the argocd-applications target and loaded rules, then compare bex:platform_gitops_expected with argocd_app_info{job="argocd-applications"}. Check the root's desired revision and the child/config revision actually running; root Synced alone does not establish that every child is healthy. Repair the demonstrated resource or configuration error through its owning workflow. This runbook does not call for forced sync, replacement, pruning or resource deletion.

Detection limit: Argo delivers this Prometheus configuration and these dashboards. If that delivery is already stalled, a rule edit is not active until the root and monitoring children reconcile; record the deployed revision and loaded rules separately from fixture-test results. These rules require a functioning Prometheus evaluator and cannot detect a whole-Prometheus outage. The committed rule tests establish failure/recovery behavior on isolated fixtures; they do not establish that production has loaded the change or bound real rollout duration.

Deployed verification (2026-09-29 05:08–05:09 UTC). Revision 32631b053 exposed 32 native and 32 independently expected Applications, all Synced/Healthy, with empty set differences; the private scrape was UP and all four rules loaded, healthy and inactive. The loaded Prometheus values/rule entries matched the revision, and the Grafana ConfigMap contained panels 16–19. After an explicit source-cache refresh, ordinary automated sync completed in 42s (root), 3s (Prometheus) and 4s (Grafana); root OutOfSync and Grafana Progressing were observed before recovery. Failure/recovery scenarios and four deliberate rule mutations were exercised only in isolated fixtures, not production.

Platform database backup telemetry

The primary-only cnpg-platform-db job covers bex-system/bex-db and auth/{kratos-db,hydra-db,openfga-db}. bex:platform_database_backup_expected records these four namespace/cluster identities independently of discovery; rendered Cluster declarations are checked against the inventory. bex:platform_database_last_archived_time keeps only timestamp samples joined to up == 1 on namespace, cluster, pod and instance, then aggregates to namespace/cluster. Matching the individual target before aggregation prevents a cached timestamp from an old primary from hiding missing telemetry on its replacement.

The two absence warnings distinguish no successful target from a successful target missing the archiver metric. Both persist for 10m; a primary switch that restores coverage before that window expires stays quiet, and recovery clears the missing condition. This is an operational allowance, not a measured failover guarantee or notification deadline; warning email follows the digest schedule. The existing raw-series PlatformDatabaseBackupStale expression remains unchanged at >26h. A timestamp of zero or CNPG's never-archived -1 sentinel is present evidence and still produces an old age; neither is coerced into missing telemetry.

The Data plane dashboard retains a row for all four expected databases in separate target and archiver-coverage tiles. Its age chart uses current-target timestamps with an explicit UNKNOWN (-1 age) fallback, so an absent database cannot disappear behind healthy siblings. PRESENT in the telemetry tile does not establish fresh WAL or restorable backups. Missing telemetry is not proof of backup data loss: follow ADR031's exporter/collector diagnosis before investigating the archiver or recovery data. The pg_stat_wal collector and pg_stat_archiver are separate; a failure in the former does not establish absence of the latter.

Registry GC recurrence and collection

The existing Zot platform-log source forwards to both retained Loki logs and a separate metrics branch. The branch accepts entries no older than 5m and emits two observation-time gauges: bex_zot_gc_failure_last_observed_timestamp_seconds for the known missing-repository-metadata GC signature, and bex_zot_log_last_observed_timestamp_seconds for any fresh Zot line. Gauges expire after 5m idle; repository identity and paths stay in logs. The private alloy job scrapes the log-shipper's HTTP metrics port every 30s, retaining these two families with bounded service=zot plus collector job, instance, node, pod attribution. The same private scrape also supplies the durable ingestion signals below.

ZotGarbageCollectionFailure compares wall time with the latest observation timestamp retained over 3h. The first fresh match warns immediately; repeated observations extend the window, and it clears 3h after the last observation. Prometheus history survives collector reload/expiry, while an old gauge reappearing cannot reopen an incident older than the window. Historical entries older than 5m cannot refresh the metric; replay within that allowance can extend observation time by up to 5m. This signal measures recurrence, not unique log events, GC runs or reclaimed bytes, and the metrics filter does not remove original Loki history.

bex:zot_log_collection_healthy independently requires an observation newer than 5m joined to successful up for the same Alloy instance. Ten minutes without that evidence warns through ZotGarbageCollectionTelemetryMissing. Cluster capacity shows recurrence, collection status and the retained failure logs next to registry capacity evidence; no failures with missing collection reads UNKNOWN, not healthy zero. The ADR060 D4 runbook distinguishes collection diagnosis from the bounded repository repair. These warnings neither prevent the upstream Zot defect nor automatically repair registry data, and runtime deployment evidence remains separate from fixture behavior.

Durable log ingestion

The Platform availability dashboard's Durable log delivery row separates Alloy final drops, Loki rejection attempts, sustained retries, collector coverage/configuration, Loki scrape coverage and WAL disk-full risk. It reuses the private alloy job and adds a private Loki scrape. Metrics contain bounded target/component/reason dimensions, not log contents, repository names or tenant identifiers. Read collection status alongside activity: a successful scrape establishes reachability, not successful or durable ingestion.

Coverage and timing. bex:log_delivery_collector_expected{node} comes from kube_node_info, independently of Alloy discovery, and covers every node including the control plane. Missing collector or inventory evidence warns after 10m; node removal removes its expectation. Invalid configuration, nonhealthy components or missing native health families under a successful expected scrape warn after 5m. Loki requires a successful scrape plus current process-start and initialized WAL-counter evidence; incomplete evidence warns after 5m. Final drops, rejected attempts and WAL disk-full activity warn without an additional persistence delay. Retry activity uses a 5m window sustained for 10m, allowing isolated bursts to recover quietly. These are operational allowances, not notification deadlines: warnings use the existing digest's separate grouping/delivery timers.

Delivery semantics. Alloy v1.11.3 (its loki.write is unchanged from v1.11.2) retries connection failures, HTTP 429 and 5xx; other failed HTTP responses cause immediate final drops. Its dropped-entry counter also covers local size/stream-limit rejection. Loki rejection counters measure rejected attempts, which can include entries later accepted after Alloy retries; they are not a count of unique permanently lost lines. Intentional loki_process_dropped_lines_total filters, including cnpg_instance_manager and Zot metrics replay suppression, stay outside these loss alerts. Quiet tenants and absent lazy writer counters do not imply failure. See the pinned Alloy client and Loki rejection dimensions.

Buffering limits. The checked-in writer has no Alloy WAL enabled: batches are held in memory. Its defaults are a 1s batch wait, 1MiB batch size, 10s request timeout, 500ms initial backoff, 5m maximum backoff and max_backoff_retries=10. Backoff and request duration determine the actual delay; these settings establish neither a guaranteed buffering duration nor a delivery deadline. A process restart or exhausted retries can lose pending lines. The writer's component health only reports invalid configuration, so a healthy process can still drop logs. See pinned defaults and writer health/WAL behavior.

Counter evidence. Activity records combine 15m increases with a first-positive check for lazily created series. That fallback requires a successful scrape now and a marker already collected 15m ago: Alloy configuration success or Loki process start time. These markers were admitted with this monitoring rollout; older Alloy up history alone cannot turn a historical positive counter into a new incident. The first 15m of target/marker observation relies on increases, so a first positive sample without a baseline can be missed. Resets, failures entirely between scrapes, or loss of Prometheus can also hide activity. Ordinary measured increases remain visible within their lookback during scrape loss. A first-positive-only observation still requires current successful collection and can become UNKNOWN on target loss. These panels report observed evidence, not exact event counts or complete loss accounting.

Isolated failure/recovery evidence. The pinned Alloy 1.11.2/Loki 3.6.7 exercise uses fixture-only 100ms batching and fixed 1s backoff. A transient 503 retried and later delivered the original line without a final drop; an 8-day-old line received a real Loki 400 rejection and a final Alloy drop while readiness stayed 200. Sustained 503s exhausted the configured budget after ten failed requests and dropped a second line. Recovery delivered a new line; queries found only the delivered lines, and an Alloy restart reset counters without replaying the dropped batch. The CNPG filter control incremented only its intentional process-drop metric. This establishes the exercised paths, not a production buffering or detection-time guarantee.

Start with the target/node named by the alert, the loaded rule and the relevant panel, then inspect without restarting collectors:

kubectl get nodes -o wide
kubectl -n monitoring get daemonset log-shipper
kubectl -n monitoring get pods -l app.kubernetes.io/instance=log-shipper -o wide
kubectl -n monitoring logs ALLOY_POD_FROM_ALERT -c alloy --since=30m
kubectl -n monitoring get pods -l app.kubernetes.io/instance=loki -o wide
kubectl -n monitoring logs statefulset/loki --since=30m
kubectl -n monitoring get endpointslices -l kubernetes.io/service-name=loki
kubectl -n monitoring get pvc

For missing collectors, compare expected nodes with Alloy placement and Prometheus targets, including control-plane nodes; another node's successful scrape cannot cover the missing one. For configuration/component failures, inspect Alloy's rejected configuration and component errors. For retries or final drops, correlate the endpoint HTTP status/reason with Loki logs, availability and configured limits; rejection can happen while readiness stays green. For WAL disk-full risk, inspect the Loki PVC's available space and write errors. WAL replay duplicates are separate and do not establish new ingestion loss.

After repair, verify current collection, new successful pushes and a fresh read through the tenant-facing path. Existing request-log and tenant-view synthetics remain independent end-to-end checks. Recovery means subsequent delivery works; it does not prove already-dropped lines were recovered. Do not blindly replay logs, clear positions/WAL files, delete data or raise ingestion limits to silence an alert. Record the deployed revision and live coverage separately from isolated fault fixtures and their timing.

Deployed verification (2026-09-29 06:32–06:34 UTC). Verified 2026-09-29 06:32:23–06:34:21 UTC at 8d1cecba9: root, Prometheus and Grafana are Synced/Healthy; complete Prometheus ConfigMap and Platform availability dashboard/panels 21–28 match the committed render, with loaded rule/configuration semantics verified. Exactly one Alloy job covers all eleven current nodes and eleven Ready DaemonSet pods; expected, UP and healthy records each contain eleven nodes with empty coverage differences. One Loki target is UP with process marker and WAL counter present; completeness and inventory are one. All nine records evaluate healthily and all eight alerts are healthy/inactive. Selected native series total 84 for Alloy and five for Loki, within the declared label allowlists. Historical ingester_error drop totals were 212 and 41,469 on two tenant collectors, with no drop/retry increases observed during the rollout; these undated lifetime totals did not create false new incidents. Zot heartbeat advanced and was 6.94s old in the final snapshot, with collection healthy. The 15m marker baseline is still warming; fixtures establish first-positive behavior after that baseline. No production log stream was deliberately interrupted.

Webhook admission pressure

bex_webhooks_delivery_admissions_total{result="admitted|capped|deduplicated"} counts the dispatcher's committed queue decisions. The vocabulary is closed; workspace ids, endpoint ids, hostnames, URLs, event ids, and payloads are never labels. bex_webhooks_delivery_capped_batch_size is a bounded histogram of the aggregate capped count from one committed feed page. An overflow also produces at most one aggregate log line per dispatch pass, not one line or evidence row per notification.

WebhookDeliveryAdmissionPressure intentionally ignores isolated cap hits and warns only when more than 100 notifications are capped in every rolling 15-minute window for ten minutes. Start with the two causes the limit is meant to contain:

Check recent deploy/resource activity for event amplification or a runaway producer.
Check webhook endpoint health and bex_webhooks_delivery_attempts_total for a retry backlog that is keeping logical notifications open.
Confirm Postgres and webhook-worker capacity before raising BEX_MAX_WEBHOOK_DELIVERIES_PER_WORKSPACE; lowering the bound sheds more webhook projections, while 0 removes the safety boundary entirely.

Capped events do not fail their source mutation and do not appear as attempted or failed delivery history. The source watermark advances in the same transaction as the aggregate admission result, so repeated alerting represents new pressure rather than replay of the same event.

Outbound webhook and push delivery

Admission pressure (above) is about the queue's bound. These two are about the wire — WebhookDeliveryFailing and PushDeliveryStale (w3/m83 t006), the rules for the two ADR052 egress channels.

Webhooks. A tenant's own broken endpoint is theirs to fix and must never page, so the signal is not the failure count but the fleet ratio: when >90% of every attempt bex made in 30 minutes failed, the common factor is bex's egress path — cluster DNS, the bex-api NetworkPolicy egress rules, outbound TLS trust, the worker itself — not N tenants simultaneously breaking N unrelated endpoints. bex_webhooks_delivery_attempts_total carries no endpoint label by design (the cardinality rule above), so "all endpoints" cannot be counted directly and a high ratio over a meaningful volume stands in for it. On a fleet with little webhook traffic one busy broken endpoint can clear the 0.02/s (≈36 attempts per 30m) floor by itself — hence warning, and hence the first debugging step is to confirm from the deliveries surface that the failures are actually spread before treating this as platform-wide.

Push. bex_push_last_success_timestamp_seconds alone would fire on every genuinely idle night, and the queue depth alone says nothing about delivery, so the rule requires both: rows waiting in send_pending/receipt_pending and no successful provider operation for over two hours. The gauge is process-local and starts at zero, so a bex-api restart makes the age trivially huge — deliberately not guarded away, because a fresh process with queued work it cannot deliver is the failure, while a process that can deliver stamps the gauge within one worker poll, far inside the 30-minute persistence window. A phone that stops buzzing looks exactly like a quiet day (ADR048), which is why this needs a rule rather than a dashboard.

Both rules use a traffic floor rather than an or vector(0) coercion: a window with no attempts produces no series at all, which reads as "nothing happened" instead of as a fake healthy zero.

Where every tenant-facing surface's signal lives

The rule pack and the tenant reads above are two of three answers to "how would we know". The third is the scheduled probe, for failures visible only from the tenant's side — a query that returns empty, a URL that 404s, an isolation invariant. ADR088 §6 carries the per-surface ledger: each tenant-facing surface classified as covered by an alert rule, covered by a scheduled probe, or waived with its reason, and no surface left implicitly uncovered.

Three tiers: critical pages, warning is a daily digest, info never pages

Every rule carries one of three severities, and the Alertmanager route in prometheus.yaml gives each a different delivery contract. The tier is the routing, so a rule's severity label is a product decision, not a hint.

tier	who owns it	delivery	timers
`critical`	on-call, now	its own email per alert group (`alertname` + `namespace`); resolved mail sent	`group_wait` 30s; re-sent every 4h while firing
`warning`	someone, this week	one digest email listing every open warning (`group_by: [severity]`, receiver `platform-digest`); no resolved mail — the next digest simply omits what cleared	a new warning joins within 30m; re-sent at most every 24h
`info`	the tenant	none — terminates at the no-config `null` receiver; visible in Alertmanager/Grafana only	—

Why the digest exists (2026-09-11): before it every warning repeated every 4h exactly like a page, and four alerts that had been firing for days — one a false positive, three real but unowned — were 24 emails a day. That is the inbox-delete-key failure mode, where real pages get deleted with the noise. A warning is a ticket, so it is delivered like one: visible every day, never six times a day.

info is the tier for customer-actionable signals that must stay visible to the dashboard/API (and in Alertmanager) but must never wake on-call: TenantCustomDomainCertNotReady and TenantCustomDomainCertExpiringSoon, where a tenant's own custom-domain DNS isn't pointing at bex (or moved after issuance), so the platform can do nothing but surface it to the tenant. Everything the operator actually owns is warning/critical and still emails.

Inhibitions keep it to one page per broken thing: CertificateNotReady silences CertificateExpiringSoon for the same Certificate, and either tenant-cert info alert silences both platform cert rules for the same Certificate — belt-and-braces over the rules' own carve-out, so a loosened selector cannot leak a tenant cert onto on-call. equal: [namespace, name] pins the pair to one object. A generic critical-inhibits-warning rule was rejected: Alertmanager treats a label missing on both sides as equal, and most bex-group alerts carry no namespace, so one namespace-less critical would have muted every namespace-less warning.

Silences persist. Alertmanager runs on a small PVC (persistence.enabled), so a silence — the standard way to acknowledge a known alert while its .pm item is open — survives a pod restart. Before this the pod was ephemeral and every restart dropped every silence, which is why none existed while four alerts fired for weeks. A StatefulSet's volumeClaimTemplates are immutable, so the resource carries Force=true,Replace=true and the Application syncs with ApplyOutOfSyncOnly=true: Argo delete-and-recreates the StatefulSet only when the StatefulSet itself drifted, never on a rule edit. Create silences through the Alertmanager API (kubectl -n monitoring port-forward svc/prometheus-alertmanager 9093, then amtool --alertmanager.url=http://127.0.0.1:9093 silence add …) with a comment naming the .pm item.

Ownership rule. An alert firing for more than three days must have either a fix in flight or a silence whose comment names its .pm note; the daily digest is the audit that keeps an unowned warning visible without making it noise. The never-issued-certificate lesson generalizes: an alert that fires on a state nobody can act on is not "a warning we tolerate", it is a bug in the rule, fixed at the rule — with a promtool test for the exact input that fooled it — and never muted at the inbox. Two more rules were re-shaped the same day for the same reason: EgressMeterTargetMissing's expected-node count now means "nodes the meter DaemonSet can schedule on" (it named one pool taint and fired continuously once a sandbox pool and a build-only node existed), and BillingWebhookDrift no longer counts a single unsigned POST to the public webhook endpoint as drift (thirteen scanner hits in three days had each paged on-call for 15 minutes); a signature-reject burst is the new BillingWebhookSignatureRejects warning.

scripts/alerts-verify.sh preserves the null and platform-digest receivers when it swaps the email receivers for its capture webhook, so its throwaway Alertmanager still loads the committed route.

The receiver secret (out-of-band, never in git)

The receiver is an email on the SendGrid relay bex already runs (the w4/m12 invite / Kratos-courier relay — smtp.sendgrid.net:587 STARTTLS, username apikey, docs/ADR012-auth.md §Email), so there's no new channel credential to mint. The smtp_smarthost/smtp_from/to are non-secret and committed in prometheus.yaml; only the SendGrid API key is out-of-band — read via smtp_auth_password_file from a Secret alertmanager-smtp (key smtp-password) in the monitoring namespace, never committed (same custody rule as the etcd/openbao-backup S3 creds). Create it once per cluster (Alertmanager stays in ContainerCreating until it exists, a deliberate one-time bootstrap like unsealing OpenBao):

# imperative (the SendGrid API key already under .env / GH-secrets custody):
kubectl create secret generic alertmanager-smtp -n monitoring \
  --from-literal=smtp-password="$BEX_SMTP_PASSWORD"

# or seal it into a committable SealedSecret (docs/ADR016-sealed-secrets.md):
scripts/seal-secret.sh monitoring alertmanager-smtp smtp-password="$BEX_SMTP_PASSWORD" \
  > deploy/gitops/base/sealed/alertmanager-smtp.sealedsecret.yaml  # then add to base kustomization

On the local mock cluster the overlay disables Alertmanager entirely (alertmanager.enabled=false) — the disposable CAPD cluster has no SMTP cred, just as it has no backup S3 creds, so the workload is dropped rather than left pending on a missing secret (mirrors the $patch: delete of the backup CronJob Applications). The server still evaluates the rule pack (visible in the Prometheus UI), and scripts/alerts-verify.sh stands up its own throwaway Alertmanager with the email receiver swapped for an in-cluster capture webhook — so no committed secret and no real mail are needed to test the loop.

Alerting heartbeat

Prometheus and Alertmanager cannot report their own death, and a notification path that silently fails looks exactly like a quiet night (a no-MX recipient once swallowed 512 alert emails, w8/m30). So the pipeline watches itself in two layers (w3/m174):

In cluster: Prometheus scrapes itself and Alertmanager. AlertmanagerNotificationsFailing, PrometheusRuleEvaluationFailures and PrometheusNotificationsDropped page on failures inside the path (they travel the same path, so they cover partial failures, not total ones).
Out of cluster: Watchdog always fires and is routed to the null receiver (it never emails). scripts/alerting-heartbeat.sh, run hourly by .github/workflows/alerting-heartbeat.yml on the self-hosted production pool, confirms the Watchdog is active in Alertmanager, that no notifications failed in the last hour, and that api.bex.co and dashboard.bex.co answer — and keeps one GitHub issue (label alerting-heartbeat) open while any of that fails. GitHub's own notifications deliver it, independent of SendGrid. This is the "external uptime check" ADR088 §1 relies on for whole-cluster outages.

Responding: a heartbeat issue naming Watchdog means Prometheus or Alertmanager is down (kubectl -n monitoring get pods); naming notification failures means delivery is broken (check the alertmanager-smtp Secret and SendGrid activity); naming the API or dashboard means a public outage.

Tenant capacity

TenantPodUnschedulable pages when a tenant serving or datastore pod (owned by a ReplicaSet, StatefulSet or CNPG Cluster in a tea-* namespace) has been unschedulable for 10m. Tenants only see their service stuck Deploying; the cause is a tenant pool at its cluster-autoscaler maximum (infra/clusterapi/overlays/hetzner-caph/cluster.yaml) or a failed scale-up. Job pods are excluded: builds have BuildQueuedTooLong, backups their staleness alerts. First checks: kubectl -n <ns> describe pod <pod> (the scheduler's reason), cluster-autoscaler logs, and the Cluster capacity dashboard's "Unschedulable tenant pods" panel.

Static hosting path

The static-server serves every tenant static site and the activator answers every wake, first-deploy, suspended and maintenance response, so each runs two replicas on the platform pool behind a PodDisruptionBudget (minAvailable: 1) with preferred anti-affinity (w3/m175). StaticHostingPathDown pages when either Deployment has no available replica for 2m — a platform-wide outage of that path; one of two missing is PlatformDeploymentNotReady's job. Both are scored from kube-state-metrics, so they cost no extra scrape. First checks: kubectl -n bex-system get deploy,pdb,pods -l 'app.bex.co/component in (static-server,activator)' -o wide; a static-server that exits at start logs "build the first static-site snapshot" when it cannot list Apps (it refuses to serve an empty host map).

Rules are tested, not just linted

The rule pack is the single source of truth embedded in prometheus.yaml. scripts/gitops-validate.sh (CI: .github/workflows/gitops.yml) extracts it and runs promtool check rules plus promtool test rules against deploy/gitops/base/rules/alerts_test.yml — unit tests pin the non-obvious expressions (backup-age fires at >26h not 25h; the 5xx ratio ignores tiny denominators; CrashLoop respects for: 10m). A regressed expression fails CI, not prod.

Verify the loop end-to-end

scripts/alerts-verify.sh (mock cluster) proves fire→notify→resolve without waiting for a real outage: it points the receiver at an in-cluster capture pod, then breaks two invariants and watches both alerts arrive and clear.

Bad rollout → kubectl -n bex-system set image deploy/bex-api api=nonexistent:bad ⇒ PlatformDeploymentNotReady (and BexApiDown) after the for: window; revert ⇒ resolved notification.
Backup staleness → apply a kube_cronjob_status_last_successful_time fixture (or temporarily shorten the rule window) ⇒ BackupCronJobStale; restore ⇒ resolved.

Observed end-to-end latency ≈ evaluation interval (default 1m) + the rule's for: + group_wait (30s) — e.g. a for: 0m alert like BackupCronJobStale notifies within ~1–2 evaluation windows; the for: 5–15m alerts add their persistence window.

Verify (mock cluster)

# deploy a sample App, then:
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/logs?resource=<app>&type=app" | jq .

# resource metrics — with BEX_PROM_URL set these are stepped history; a ranged
# query returns one point per resolution step per instance:
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/metrics/cpu?resource=<app>&percentage=true" | jq .
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/metrics/memory?resource=<app>&startTime=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)&endTime=$(date -u +%Y-%m-%dT%H:%M:%SZ)&resolutionSeconds=60" \
  | jq '.[0].values | length'   # ≈60 points over the hour (where data exists)
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/metrics/instance-count?resource=<app>" | jq .

FilesExpand file tree

ADR010-observability.md

Latest commit

History

ADR010-observability.md

File metadata and controls

Observability — logs and metrics

Logs

The tail reads pod logs, not Loki

Durability & retention

Cluster enablement

Tenant log content (w8/m48, 2026-10-02)

Log labels (and the cardinality budget)

Log filters

REST surface

Log types

Live tail

Render compatibility

RBAC

Request correlation: Rndr-Id and requestID (w8/m125)

Service incarnations (w8/m124)

Metrics

REST surface

Render compatibility

RBAC

Cluster enablement

Production replay budget (w4/m144, 2026-10-02)

Extended metrics: autoscale-target, disk, DB connections/replication-lag (w3/m10)

Platform alerting (Alertmanager)

The rule pack

Tenant-node filesystem headroom

Platform GitOps delivery

Platform database backup telemetry

Registry GC recurrence and collection

Durable log ingestion

Webhook admission pressure

Outbound webhook and push delivery

Where every tenant-facing surface's signal lives

Three tiers: critical pages, warning is a daily digest, info never pages

The receiver secret (out-of-band, never in git)

Alerting heartbeat

Tenant capacity

Static hosting path

Rules are tested, not just linted

Verify the loop end-to-end

Verify (mock cluster)

Request correlation: `Rndr-Id` and `requestID` (w8/m125)