config

Server configuration

plural-metrics-server --config <path> reads YAML into the configuration below. The default path is config/metrics.yaml. Unknown fields are rejected in the server, listener, write, authentication, JWT, and namespace sections. The checked-in metrics.example.yaml is a valid, commented standalone configuration with alternatives for every tagged variant.

plural-logs-server --config <path> uses the same shared server, storage, write, sharding, authentication, and namespace conventions for Loki-compatible logs. Start from logs.example.yaml; the container image defaults to /app/config/logs.example.yaml. Logs listens on HTTP port 3100 and internal gRPC port 9091 in the example configuration.

plural-traces-server --config <path> serves Tempo-compatible trace reads and OTLP/Zipkin writes. Start from traces.example.yaml. It listens on HTTP 3200, internal writer gRPC 9092, and OTLP TraceService gRPC 4317. OTLP HTTP is available at /write/ns/{namespace}/v1/traces; OTLP gRPC takes the namespace from x-scope-orgid. Zipkin v2 JSON is accepted at /write/ns/{namespace}/api/v2/spans. Jaeger collector and Thrift transports are not currently implemented; /write/ns/{namespace}/api/traces returns 501 and does not pretend that JSON is Jaeger Thrift.

pseudofs-server --config <path> serves multiple tenant-isolated virtual filesystems from one SlateDB writer over public gRPC. Start from pseudofs.example.yaml. It listens on 9093, enables gRPC health and reflection, and requires a validated tenant identifier on every filesystem request. Tenants are selected dynamically rather than configured as an allowlist; PseudoFS intentionally has no sharding or application authentication.

Traces reads are rooted at /read/ns/{namespace}: trace-by-ID v1/v2, TraceQL /api/search, v1/v2 tag names and values, and /api/echo. TraceQL metrics routes return 501 because metrics execution is deliberately outside Traces' current scope. JSON is the default trace response; request a protobuf media type in Accept for protobuf. Reader mode disables writes, writer mode disables reads, and standalone mode enables both.

Traces supports standalone and static shard ownership. The checked-in server reports Kubernetes Traces sharding as unsupported at startup rather than silently running with incorrect ownership; use static ownership for split deployments until the Traces Kubernetes lifecycle is added.

Defaults apply when a field or section is omitted. Fields described as required must be present when their containing section or tagged variant is present.

Live reload

Metrics, Logs, and Traces check their config file every 10 seconds and apply a changed revision without restarting. The operator mounts the config Secret as a directory, which the kubelet refreshes in place, so a config change reaches running pods within about a minute and does not roll them. These settings take effect immediately:

• namespaces: adding or removing namespaces and changing their auth.
• auth.unauthenticated and auth.global. Credential files are reread on every request, so rotating a referenced secret needs no reload at all.
• cache_warmer, including continuous. A reload restarts the warmer without rerunning the startup pass.

Every other change, including namespace usage_reporting_endpoint, auth.jwt, and auth.internal, takes effect only after the process restarts. The server logs the affected sections and sets telemetry_config_restart_required{product} to 1 until a restart or a revert. An invalid revision is rejected and the running config stays; telemetry_config_reloads_total{product,outcome="applied|invalid"} counts both. PseudoFS does not reload, so the operator still rolls its pod when its config changes.

Server and listeners

• mode: server role. Valid values are standalone (default), writer, and reader. standalone serves reads and writes and must use sharding.backend: standalone. writer exposes external write routes plus the internal writer gRPC service; reader exposes only read routes. Split writer/reader deployments normally use static or kubernetes sharding.
• listeners.http: HTTP bind socket. Default 0.0.0.0:8080.
• listeners.grpc: internal writer gRPC bind socket. Default 0.0.0.0:9090.
• path_prefix: optional prefix for public read and write APIs, such as /metrics. It must start with / and must not end with /. The default is empty.

Read APIs are under {path_prefix}/read/ns/{namespace} and write APIs are under {path_prefix}/write/ns/{namespace}. Read routes are /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, /api/v1/label/{name}/values, /api/v1/metadata, and /federate. Write routes are /api/v1/write (Prometheus remote write) and /v1/metrics (OTLP/HTTP protobuf). /-/healthy, /-/ready, and /metrics are not namespace-prefixed. Every Metrics, Logs, and Traces process exposes its internal metrics crate recorder at GET /metrics using the Prometheus text exposition format. Scrape the product's HTTP service and port directly; the endpoint includes cache-warmer, SlateDB, query, ingestion, and process-role metrics.

storage

Metrics always uses SlateDB. If the whole section is omitted, it defaults to path: data, an InMemory object store, no settings file, and no caches. If storage is present, path and object_store are required.

• storage.path: object-key prefix for Metrics data. Sharded deployments place one multi-namespace SlateDB below each shard-NNNN child; namespace isolation is encoded in database keys. Default data.

• storage.settings_path: optional path to a SlateDB TOML, JSON, or YAML settings file. If omitted, SlateDB loads its normal SlateDb.toml, SlateDb.json, or SlateDb.yaml files and SLATEDB_ environment overrides; load errors fail startup. See config/SlateDb.example.toml; Zstd is recommended for object-store deployments with string-heavy discovery catalogs, while LZ4 trades compression ratio for lower CPU overhead. Metrics layers the file over its own defaults rather than SlateDB's, so any key the file sets wins: l0_sst_size_bytes 16 MiB (SlateDB: 64 MiB) and compactor_options.scheduler_options.min_compaction_sources "3" (SlateDB: "4"). They keep queries over recent data near compacted speed at the cost of more object-store puts and compaction work; see Metrics configuration.

• storage.object_store: required object-store variant:

  • type: InMemory: process-local, nonpersistent storage; no additional fields.
  • type: Local: local filesystem storage; requires path.
  • type: Aws: S3 storage; requires region and bucket. Optional fields are endpoint, allow_http, and virtual_hosted_style. Credentials are resolved from standard AWS environment variables and ambient providers such as web identity, ECS, and EC2 metadata.
  • type: Azure: Azure Blob Storage; requires account and container. Optional fields are endpoint and allow_http. Account keys, SAS tokens, bearer tokens, service principals, workload identity, managed identity, and the Azure CLI are supported through the standard AZURE_* object-store environment variables.
  • type: Gcp: Google Cloud Storage; requires bucket and optionally accepts base_url. Service-account JSON, bearer tokens, and application default credentials are supported through the standard GOOGLE_* object-store environment variables.

  For Aws, Azure, and Gcp, identical range reads that are in flight at the same time share one request across every shard, database, query, and cache-warming pass in the process. Nothing is cached at this layer; telemetry_object_store_coalesced_gets_total{outcome="issued|shared"} counts requests sent versus joined.

• storage.block_cache: optional SlateDB SST data-block cache.

• storage.meta_cache: optional SlateDB index/filter/stats cache. Either cache may use either cache variant; leaving one side absent disables caching for that block class.

Every storage shard a process opens shares one block cache and one meta cache, so their capacities are per-process budgets rather than per-shard ones. When both are FoyerHybrid, they must use different disk_path directories.

Cache variants are:

• type: FoyerMemory
  • capacity: required capacity in bytes.
  • shards: optional shard count. When absent, Foyer derives it from available CPUs.
• type: FoyerHybrid
  • memory_capacity: required memory-tier capacity in bytes.
  • disk_capacity: required disk-tier capacity in bytes.
  • disk_path: required cache directory.
  • write_policy: WriteOnInsertion (default; send inserted entries to disk) or WriteOnEviction (send entries to disk when evicted from memory). These values are case-sensitive.
  • flushers: large-engine flush thread count. Default 4.
  • buffer_pool_size: optional bytes reserved for the large-engine flush pipeline. The effective default is memory_capacity / 32; each flusher double-buffers, so actual allocation is about twice this value.
  • submit_queue_size_threshold: queued bytes before cache entries are dropped. Default 1073741824 (1 GiB).

retention_seconds is optional Metrics retention. Each time bucket's records are written with a SlateDB TTL of bucket start plus the retention window, so compaction drops the whole bucket at once. Unset (the default) keeps data forever.

reader_cache_capacity controls each shard reader's Moka cache of loaded time-bucket query readers. The current implementation does not install a weight function, so the unit is entries, not bytes. Default 268435456.

cache_warmer

The startup cache warmer applies to Metrics, Logs, and Traces readers and standalone servers. Writer-only processes skip warming. Errors are logged and do not permanently block readiness. Warming consumes permits from the same pod-wide storage I/O budget as queries. Metadata-only warming is restricted to SlateDB filters and SST indexes, which use the dedicated metadata cache and cannot evict payload blocks from the data cache. It warms the newest SSTs first and stops once what it warmed reaches metadata_budget_percent of the metadata cache, counting the SSTs that did not fit; later passes forget, and evict from the cache, what they warmed of SSTs compaction, rollups or retention removed. With no metadata cache configured nothing is cached, so metadata-only warming does nothing.

Metrics and Traces warm by default, in the background: the server is ready at once and queries during the pass find the cache partly warm. Logs keeps the earlier default of no startup warming, and when enabled holds readiness until the pass ends. A cache_warmer section that sets only some keys keeps the product's defaults for the rest (the configs reject unknown keys, so older configs keep parsing and new keys need no migration).

• cache_warmer.enabled: enables startup warming. Default true for Metrics and Traces, false for Logs.

• cache_warmer.block_readiness: hold readiness false until the startup pass ends or times out. Default false for Metrics and Traces, true for Logs.

• cache_warmer.metadata_budget_percent: the share of the metadata cache warming may fill, 0 to 100. Default 75, which leaves the rest for metadata queries fetch.

• cache_warmer.warm_range_seconds: recent time range whose filters and indexes are warmed. Default 0, which warms all live data: retention bounds what is live, and metadata_budget_percent bounds what is held, newest first. A set value also bounds payloads unless payload_warm_range_seconds is set, as it did before payloads had their own range.

• cache_warmer.payload_warm_range_seconds: recent time range whose payloads are warmed when include_payloads is set. Default 0, which uses warm_range_seconds when that is set and otherwise all live data, bounded by payload_budget_percent. Servers older than this key reject it.

• cache_warmer.payload_budget_percent: the share of the data cache payload warming may fill, 0 to 100, newest first: of the disk tier of a FoyerHybrid block_cache, otherwise of its memory. The warmer sizes itself from the cache capacities, which the operator derives from the pod's memory and volume. Older SSTs' payloads stop counting when newer ones need the room, and SSTs whose payloads do not fit still warm their metadata. Default 50. Servers older than this key reject it.

  Payload warming never fills memory: with a hybrid cache, warmed blocks are written to the disk tier only, disk hits during warming are not promoted into memory, and warming waits for the disk tier to write what is queued rather than overflow foyer's submit queue (which drops entries). SlateDB reads all of an SST's data blocks before caching them, so warming also holds no more than a quarter of the data cache's memory capacity (at least 16 MiB) of data blocks in flight; one SST larger than that still warms, alone.

• cache_warmer.timeout_seconds: maximum time warming may hold readiness false, with block_readiness. Default 30; timeout is fail-open.

• cache_warmer.concurrency: maximum concurrent SST warm operations and number of permits reserved from the pod-wide storage I/O budget. Default 2; must be greater than zero when warming is enabled.

• cache_warmer.include_payloads: also warms metric samples, log pages, and trace pages. When false, only SlateDB filters and SST indexes are warmed. Default false.

Readers learn about new L0 SSTs and compaction outputs from manifest polls but only fetch their blocks on the first query that touches them, so the newest data is always read cold from object storage. Continuous warming closes that gap: every interval it lists the live SSTs in the recent window and warms only those it has not warmed before, so each pass costs roughly the bytes flushed or compacted since the previous one. When startup warming is disabled, the first pass only records the SSTs already live, so enabling it never starts with a bulk warm of the whole window. Passes use cache_warmer.concurrency and the same storage I/O budget as startup warming, and never affect readiness.

• cache_warmer.continuous.enabled: enables continuous warming. Default true.
• cache_warmer.continuous.interval_seconds: seconds between passes. Default 10; must be greater than zero when enabled.
• cache_warmer.continuous.warm_range_seconds: as cache_warmer.warm_range_seconds, for new SSTs. Default 0, all live data.
• cache_warmer.continuous.payload_warm_range_seconds: as cache_warmer.payload_warm_range_seconds, for new SSTs. Default 0, which is 7200 unless continuous.warm_range_seconds is set.
• cache_warmer.continuous.include_payloads: also warms metric samples, log pages, and trace pages of the SSTs inside the payload range. Default true.

The metadata range used to default to 7200 for both passes. It predates the metadata budget, which now bounds memory however wide the range is; a wider range only means older SSTs compete for the budget, newest first, and what does not fit is skipped and counted. Configs that set warm_range_seconds explicitly keep their meaning for metadata and payloads alike.

write

• write.durability: acknowledgement guarantee. Valid values are applied, written, and durable; default applied.
  • applied: accepted into the in-memory delta, but not necessarily visible to snapshot-backed queries.
  • written: moved into SlateDB mutable state and visible to a fresh snapshot, but not yet guaranteed across restart.
  • durable: flushed to the configured object store before acknowledgement.
• write.flush_interval_seconds: seconds between durable flushes of active writers. All products default to 10; 0 disables the periodic task. This bounds persistence and split-reader visibility for applied and written requests. Shutdown also flushes open writers.
• write.remote_concurrency: maximum concurrent shard batches forwarded to owners. Default 16.
• write.remote_retries: retries after the initial internal gRPC attempt when ownership generation is stale. Default 2.

sharding

• sharding.shards: storage-shard count for standalone and static backends. Default 1; must be greater than zero and remain identical across processes and restarts. Kubernetes derives the desired storage-shard count from writer StatefulSet replicas instead.
• sharding.io_concurrency_limit: fixed concurrent storage I/O operations per pod. Default 128; it does not scale with the number of open shards. must be greater than zero. Increase it when I/O latency leaves shard operations idle.
• sharding.backend: standalone (default), static, or kubernetes.

standalone has no additional fields and assigns every shard to the process.

static requires:

• owner_id: this process's owner ID; it must match an owners[].id.
• owners: fixed owner list. Every entry requires id, numeric ordinal, internal gRPC endpoint, and the half-open range [start_shard, end_shard). Ranges must be nonempty, contiguous, nonoverlapping, start at zero, and exactly cover shards.

kubernetes discovers writer membership from a StatefulSet, stores assignments in a ShardMap CR, and uses Leases for coordinator and shard ownership. The server must be built with the kubernetes feature (enabled by default) and have namespace-scoped RBAC. Fields are:

• database: database name. Every Lease is labeled telemetry.plural.sh/metrics=<database>, and lease watches select on that label. Must be a valid label value. Default metrics.
• namespace: Kubernetes namespace. Default default.
• stateful_set: writer StatefulSet name. Default metrics.
• headless_service: writer headless Service used for owner DNS. Default metrics-headless.
• owner_port: internal gRPC port. Default 9090.
• shard_map: authoritative ShardMap resource name. Default metrics-shard-map.
• coordinator_lease: coordinator Lease name. Default metrics-shard-coordinator.
• shard_lease_prefix: prefix for per-shard Lease names. Default metrics-shard.
• lease_duration_seconds: shard/coordinator lease duration in seconds. Default 15.
• renew_interval_seconds: shard ownership renewal interval in seconds. Default 5.

The local Kubernetes owner ID is POD_NAME when set, otherwise <stateful_set>-${POD_ORDINAL:-0}. The coordinator balances storage-shard ownership as StatefulSet membership changes. The desired storage-shard count is the StatefulSet replica count. The ShardMap snapshot contains both ownership and versioned 128-bit hash ranges and is updated with Kubernetes resourceVersion compare-and-swap. Writers and readers watch the ShardMap stream for coherent generation updates. Writers acquire a stable Lease for each assigned shard and watch Lease releases for immediate handoff. Coordinator candidates also watch the coordinator Lease for prompt failover.

Secrets and authentication

A secret value supports exactly one source:

{ source: literal, value: development-only }
{ source: env, name: METRICS_PASSWORD }
{ source: file, path: /var/run/secrets/metrics/password }

env reads the named environment variable and file reads UTF-8 text from the path. Trailing CR/LF characters are removed; an empty result is rejected. Secret values are redacted from debug output. Prefer environment or read-only mounted files in production.

auth.unauthenticated defaults to false. Therefore every namespace read/write HTTP request requires an applicable Basic credential or valid JWT unless anonymous access is explicitly enabled with auth.unauthenticated: true. Empty credential lists never imply anonymous access.

auth.global.read and auth.global.write are credential lists accepted for every namespace. Each namespaces[].auth.read and .write list adds credentials for that namespace and permission. The only configured credential type is HTTP Basic:

- type: basic
  username: prometheus
  password: { source: file, path: /var/run/secrets/metrics/password }

Global and namespace Basic credentials are alternatives, not cumulative requirements. When auth.unauthenticated is false, a request must present either an applicable Basic credential or a valid Bearer JWT. When it is explicitly true, anonymous namespace access is allowed even if Basic or JWT credentials are also configured.

auth.internal is an optional secret used only for writer-to-writer gRPC. When set, callers send and owners require authorization: Bearer <token>. Configure the same value on every writer and reader that may forward writes. This token does not authorize external HTTP routes. When omitted, the internal gRPC method has no token check.

External JWT/JWKS

auth.jwt enables Bearer JWT verification for every namespace:

• jwks: required source:
  • { source: file, path: ... }: read once during startup. The path must be nonempty; file changes are not reloaded.
  • { source: url, url: ... }: fetched during startup and then refreshed lazily by requests after refresh_interval_seconds. A token with an unknown kid triggers an immediate refresh, rate-limited to the same interval. Failed refreshes keep the last valid key set.
• issuer: optional exact iss requirement. When set, the claim is required.
• audience: optional aud requirement. When absent, audience validation is disabled.
• refresh_interval_seconds: URL refresh and unknown-key retry interval. Default 300; must be greater than zero.
• request_timeout_seconds: HTTP timeout for URL retrieval. Default 5; must be greater than zero.

The JWKS must contain a nonempty keys array. Every key needs a unique nonempty kid, a supported alg, and valid JWK key material. Supported algorithms are HS256, HS384, HS512, ES256, ES384, RS256, RS384, RS512, PS256, PS384, PS512, and EdDSA. The JWT header must contain a matching kid and alg.

JWT claims:

• exp: required expiration timestamp and validated.
• nbf: optional not-before timestamp and validated when present.
• namespace: required Rust regular expression matched against the requested configured namespace. Use anchors such as ^tenant-a$ for an exact match.
• permission: required exact value read or write.
• iss and aud: required and validated only when their corresponding configuration fields are set.

namespaces

namespaces defaults to one namespace named default and must not be empty. That namespace is protected by the default auth.unauthenticated: false HTTP policy. Entries require a unique name; names must be 1–255 bytes and cannot contain NUL or ASCII control characters. Only listed namespaces are opened. Requests for any other name return 404.

Each namespace has isolated storage and query state. Its APIs use the literal configured name in /read/ns/{namespace} and /write/ns/{namespace}, optionally below path_prefix, while its storage prefix uses the deterministic namespace hash described above. Authentication is evaluated separately for read and write access.