scaling.md

Capacity planning and scaling

This page translates SlateDB's public key/value benchmarks into provisional capacity guidance for Metrics, Logs, and Traces. The values below are planning envelopes, not measured Telemetry service limits or service-level objectives. Run the product-shaped benchmarks described below on the intended object store and instance type before setting production limits.

Executive guidance

Start a production writer with 8 vCPU, 32 GiB of memory, and 10 Gbit/s network capacity in the same region as its object store. Keep sustained ingestion below 65% of the measured saturation point so compaction, traffic bursts, shard handoff, and object-store tail latency have headroom.

Until native product benchmarks replace these estimates, use the following provisional envelope for one writer:

• Logs: 10–30 MiB/s of uncompressed log bodies, approximately 0.8–2.5 TiB/day.
• Metrics: 250,000–500,000 samples/s for stable series; series churn can lower this substantially.
• Traces: 20,000–60,000 spans/s when a span averages approximately 500 bytes.

These ranges assume batched ingestion, default applied durability, limited writer-side reads, and an object store in the same region. They are deliberately below SlateDB's best public results. Do not multiply the upper bound by the writer count without testing shared object-store, compactor, and network limits.

Increase writer replicas when any one writer sustains more than 65% of its validated byte, CPU, memory, or compaction capacity. Writer replicas are storage shards, so an increase appends a routing epoch: records timestamped after the next cutover spread across the larger shard count, while existing data stays on its original shards. Scale-down remains blocked.

Scale readers independently from writers. Query concurrency, cache hit rate, scan width, and object-store GET latency determine reader count and are not covered by the ingestion estimates on this page. With reader groups, readers scale per storage shard: see Reader groups and cost.

What limits a SlateDB writer

SlateDB writes its WAL and LSM data to object storage. Local disk is an optional, disposable cache tier; it is not the primary durability path. The important writer limits are:

1. CPU used by encoding, indexing, compression, and compaction.
2. Network bandwidth generated by WAL, L0, and compaction traffic.
3. Object-store PUT latency, throughput, and request limits.
4. Memory for unflushed data, active product state, and caches.
5. L0 or compaction backlog that causes write throttling.

Fast local NVMe helps Foyer read caching and mixed read/write workloads, but it does not remove object-store durability latency.

SlateDB groups many application writes into WAL and L0 objects. One telemetry request does not normally equal one S3 PUT. Application batching still matters because it reduces product-level locks, page fragmentation, metadata updates, and indexing work before data reaches SlateDB.

Public SlateDB reference points

The SlateDB 0.15 release benchmarks use 20-byte keys, 400-byte values, 64 closed-loop clients, S3 in us-east-1, and a large ARM runner. The dataset for steady-state workloads contains 300 million records and approximately 120 GiB of logical data.

The sustained-ingest workload reached approximately:

• 299,000 puts/s and 120 MiB/s of logical application writes.
• 588 MiB/s of object-store PUT traffic and 464 MiB/s of GET traffic.
• 8.6 CPU cores and 28 GiB RSS on average.

This is an optimistic ceiling: the database begins empty, keys are unique and sequential, and WAL is disabled for that workload. Its roughly 5x object-store PUT amplification also shows why network and compaction must be measured.

The update-heavy workload is closer to a live indexed database. It uses WAL, runs 95% updates and 5% point reads, and reached:

• 84,000 puts/s and 34 MiB/s of logical application writes.
• 272 MiB/s of object-store PUT traffic, about 8x logical write bandwidth.
• 5.3 CPU cores and 20 GiB RSS on average.
• 246 ms average time from write acceptance to remote durability.

The balanced workload reached only 6,200 puts/s alongside 6,200 gets/s because remote cache misses hold back a task that mixes reads and writes. This supports Telemetry's separation of writer and reader roles.

SlateDB's tuning guidance quotes expected object-store write latency of 50–100 ms for S3 Standard, 5–10 ms for S3 Express One Zone, and 5–20 ms for MinIO. Synchronous small writes therefore require group commit or application batching. written ingestion does not wait for each object-store round trip; durable ingestion does.

Shared write-buffer behavior

Metrics, Logs, and Traces use the same bounded, per-storage-shard write coordinator. A request is validated and merged into a live in-memory delta; the coordinator freezes deltas by size, time, or an explicit durability barrier. Logs coalesces entries by namespace, segment, and stream fingerprint. Traces coalesces traces by namespace and segment. This allows separate client requests to share product page and index work.

The requested durability level controls acknowledgement:

• applied acknowledges after the write has entered the live delta. It gives the most opportunity for cross-request coalescing but can lose acknowledged data if the process exits before a flush. This is the default for telemetry throughput.
• written freezes through the request's epoch and waits until its coalesced batch has entered SlateDB's writable state. It does not force a remote object-store flush.
• durable also waits for SlateDB's object-store flush. Its latency therefore includes object-store PUT latency.

All three products default write.flush_interval_seconds to 10 seconds. For applied and written writes, read-only replicas normally observe data within roughly 0–10 seconds plus manifest polling and flush latency; size-triggered or explicit flushes can make it visible sooner. This durable interval is separate from buffer_flush_interval_milliseconds, which controls when Logs and Traces freeze and apply their live in-memory deltas.

The queue is deliberately bounded. A full queue produces protocol-specific backpressure (429 plus Retry-After for HTTP and UNAVAILABLE for OTLP or Jaeger gRPC) rather than unbounded memory growth. Storage or drain unavailability returns 503 or UNAVAILABLE; clients should reroute or retry. OpenTelemetry Collector exporters and Grafana Alloy retry these responses with backoff by default. Grafana Alloy also retries Loki and Prometheus remote-write 429 responses by default. Stock Prometheus requires remote_write.queue_config.retry_on_http_429: true; without it, Prometheus drops a batch rejected with 429. All conforming remote-write senders retry the 5xx responses used for transient storage and drain failures. Legacy applications that send directly with native Zipkin or Jaeger client reporters commonly do not retry rejected batches at all. Put an OpenTelemetry Collector or another durable agent between those applications and Telemetry when overload loss is unacceptable; changing the response code cannot make those legacy reporters replay a dropped batch.

Plan coordinator memory per open storage shard as:

live delta + up to two frozen deltas + queued request payloads

buffer_size_threshold_bytes bounds each delta approximately, not total process memory. buffer_queue_capacity counts queued commands rather than bytes. Lower buffer_flush_interval_milliseconds reduces applied latency and drain work but also reduces batching. Raise the size threshold only after measuring process RSS and flush latency.

During a shard handoff, the drain gate waits for already-admitted writes to finish, rejects later local writes, then runs drain → flush → close → lease release. The flush advances buffered writes through durable storage before the next owner opens the shard, so every acknowledged pre-drain write is included.

Product write behavior

Metrics

Metrics first combines samples in per-bucket deltas. written flushes those deltas into SlateDB's writable state without waiting for object storage, and a periodic flush establishes remote durability and reader visibility. Repeated samples for stable series reuse schema and posting state.

Metrics capacity depends on both samples/s and active-series behavior:

• Stable series primarily exercise sample encoding and delta merging.
• New or changed series also create schema and posting records.
• High churn can become the limit even when sample throughput is moderate.
• Classic histograms create one series per bucket, plus _sum and _count. Native histograms (remote-write histograms and OTLP exponential histograms) are stored as one series with larger samples.

Logs

Logs turns each frozen delta into coalesced SlateDB WriteBatch operations. Default applied requests can coalesce across the configured time and size window. written requests force a coordinator barrier but not an object-store flush. The common periodic durable flush makes complete accepted batches visible to read-only replicas.

Large batches let Logs fill its approximately 1 MiB pages, resolve a stream once for more entries, and combine posting updates. Small requests increase lock, sequence, page, and index work per log byte. Stream cardinality and churn should therefore be tested independently from byte throughput.

Traces

Traces also coalesces pages, locators, metadata, and posting fragments before committing atomic SlateDB WriteBatch operations, followed by periodic durable flushes that establish reader visibility. Trace batches should be large enough to fill approximately 1 MiB pages without turning an entire object-store buffer object into one oversized transaction.

Traces capacity is sensitive to average span size, spans per trace, attribute count, indexed-attribute cardinality, and continuation writes to existing traces.

Batching and an optional ingest buffer

Clients should normally send or coalesce 1–4 MiB per-shard writes with a 10–50 ms maximum aggregation delay. This is large enough to amortize product overhead while bounding memory, tail latency, and handoff drain time.

An OpenData buffer in front of Telemetry can durably absorb bursts and merge many small requests before calling a writer. To improve database throughput, the consumer must decode entries, group them by product, namespace, and routing shard, and submit larger product-level writes. Replaying every original request individually preserves availability benefits but leaves most database overhead unchanged.

The buffer does not increase SlateDB's physical compaction or network ceiling. It can increase useful telemetry throughput by reducing application-level write amplification. For example:

raw telemetry capacity
    = measured SlateDB logical-write capacity
      / measured application write amplification

Reducing application amplification from 3x to 1.5x doubles useful telemetry throughput at the same SlateDB logical-byte ceiling.

Exactly-once buffered ingestion requires care. A single queue sequence that fans out to several physical shards cannot be committed atomically without a distributed transaction. Prefer buffer partitions that follow the storage-shard hash ranges of the current routing epoch so each queue sequence targets one physical database, or accept at-least-once delivery and make product writes idempotent.

Translating throughput into production units

Logs: logs/day and entries/s

Grafana's Loki deployment guidance positions monolithic mode for approximately 20 GB/day and simple scalable mode close to 1 TB/day before microservices mode is preferred. Red Hat's OpenShift Loki profiles define production tiers of:

• 100 GB/day with 14 requested vCPU and 31 GiB memory cluster-wide.
• 500 GB/day with 34 requested vCPU and 67 GiB memory.
• 2 TB/day with 54 requested vCPU and 139 GiB memory.

Those figures include replicated, distributed Loki components and query capacity, so they are context rather than an apples-to-apples writer comparison. Telemetry stores one unreplicated SlateDB shard per writer and scales readers separately.

Use an approximately 650-byte average event for an initial mixed workload: 70% 250-byte events, 25% 1 KiB events, and 5% 4 KiB events. With that mix:

• 500 GB/day is approximately 5.8 MB/s and 8,900 entries/s.
• 1 TB/day is approximately 11.6 MB/s and 17,800 entries/s.
• 2 TB/day is approximately 23.1 MB/s and 35,500 entries/s.
• 4 TB/day is approximately 46.3 MB/s and 71,000 entries/s.

The provisional single-writer target of 10–30 MiB/s corresponds to roughly 16,000–48,000 entries/s for this event mix. It should cover the OpenShift 500 GB/day tier and is intended to approach the 2 TB/day tier with efficient batching. Only a product benchmark can confirm that expectation.

Grafana recommends 5–10 stable labels, reasonably sized lines, and explicit tenant rate limits. Benchmark at least 1,000, 10,000, and 100,000 active streams because stream churn and posting cardinality can force scaling before bytes/s.

Metrics: samples/s and active series

Grafana's Mimir capacity guide estimates distributor resources at 1 CPU and 1 GiB memory per 25,000 samples/s. It estimates ingester resources at 1 CPU, 2.5 GiB memory, and 5 GB disk per 300,000 in-memory series. Mimir normally replicates ingested series, while Telemetry currently relies on the shared object store and does not replicate shards in-cluster.

A production-like node-exporter workload has approximately 800 series per target. At a 15-second scrape interval:

• 1,000 targets produce approximately 53,000 samples/s and 800,000 series.
• 5,000 targets produce approximately 267,000 samples/s and 4 million series.
• 10,000 targets produce approximately 533,000 samples/s and 8 million series.
• 20,000 targets produce approximately 1.07 million samples/s and 16 million series.

The provisional single-writer target is 250,000–500,000 samples/s for stable series. Compared with Mimir's distributor ratio, that is 10–20 CPU-equivalents of protocol ingestion before accounting for Mimir's separate replicated ingesters. Telemetry's 8-vCPU/32-GiB starting recommendation is plausible only because Metrics combines distribution and storage differently; it remains an estimate to validate, not evidence that it is more CPU efficient.

Run separate steady-state, 1% hourly churn, and 10% hourly churn tests. Report active series and new series/s alongside samples/s. Prometheus remote-write wire size varies from a few bytes to tens of bytes per sample depending on protocol version, labels, and batching, so network bytes/sample is not a stable capacity unit by itself.

Traces: spans/s

Tempo's ingestion guidance uses an approximately 500-byte typical span. Its documented 15 MB/s per-distributor default corresponds to about 30,000 spans/s, and three distributors with local rate limiting provide about 90,000 spans/s of aggregate admission capacity.

At the same 500-byte average:

• 10 MiB/s is approximately 21,000 spans/s.
• 20 MiB/s is approximately 42,000 spans/s.
• 30 MiB/s is approximately 63,000 spans/s.

The provisional Traces target of 20,000–60,000 spans/s per writer therefore spans roughly one to two Tempo default distributor admission envelopes. Tempo uses separate downstream components and different durability, replication, and query architecture, so this comparison is a workload translation rather than a claim of equivalent end-to-end capacity.

Benchmark small and large traces separately. Keep average bytes/span, spans per trace, attributes/span, indexed attributes, and concurrently live traces fixed and reported.

Provisional writer profiles

Use these only to select the first benchmark or staging deployment:

• Small, 4 vCPU and 16 GiB: up to 250 GB/day logs, 100,000 samples/s, or 10,000 spans/s.
• Standard, 8 vCPU and 32 GiB: target 0.5–1 TB/day logs, 250,000–500,000 samples/s, or 20,000–40,000 spans/s.
• Large, 16 vCPU and 64 GiB: test toward 2 TB/day logs, 500,000–1 million samples/s, or 40,000–60,000 spans/s.

Provide at least 1 Gbit/s for the small profile and 10 Gbit/s for standard and large profiles. SlateDB's public results show object-store traffic 5–8 times logical application write bandwidth during compaction, so application ingress alone understates required network capacity.

Do not use these profiles as Kubernetes defaults until the product benchmarks pass on S3 Standard and the intended cache configuration.

Reader groups and cost

Kubernetes sharded deployments run one reader group per storage shard by default. For S storage shards and r replicas per group, the reader tier is S × r pods, and each pod opens one shard. Without groups, R reader pods each open all S shards.

What changes for the same per-shard availability (r = R):

• Pods and fixed memory. Groups run S times as many reader pods, so the process baseline (runtime, connection pools, and the configured block and meta cache budgets) is paid S × r times instead of R times. Per-shard SlateDB reader state and WAL replay buffers are the same in total (r × S copies either way), but each group pod holds only one.

• Cache efficiency. A group pod's cache serves one shard's working set rather than the whole dataset's. A per-pod cache of about 1 / S of the former size gives a similar hit rate, which offsets most of the extra baseline. Size storage.block_cache for one shard's hot data, not the dataset's.

• Object-store requests. Manifest polling scales with the number of (pod, shard) pairs, r × S, which is unchanged at r = R. Cache-miss GETs per shard scale with that shard's group size, and better hit rates usually lower them.

• Network transfer. A query moves the partial results of S - 1 remote shards to the coordinating reader: Metrics moves the selected series' chunk-encoded samples, Logs moves matched lines or per-shard aggregates, and Traces moves matching traces and hits. Reading the local shard costs nothing. When groups span availability zones, most remote traffic crosses a zone, which cloud providers bill per GiB in each direction (for example $0.01/GiB each way between AWS zones in one region):

  monthly cross-zone cost ≈
      queries per month
      × average remote response bytes per query
      × (S - 1) / S
      × fraction of peers in another zone (≈ 2/3 across three zones)
      × billed price per GiB (both directions)
  

  Metrics range queries over many series move the most bytes, because PromQL runs on the coordinator. Aggregating Logs metric queries move the least.

• CPU. Shard work runs in parallel across groups rather than in one process, which lowers latency for wide queries. Total CPU is about the same plus protobuf encoding and decoding of remote results.

• Hedging. A unary call that has not answered after hedge_after_ms is started on a second replica, so the slowest calls can run twice. Streaming reads, which carry nearly all the bytes and memory, are never hedged. Raise the delay or set it to zero to trade tail latency for less duplicate work.

Groups pay off when the dataset, shard count, or query load is large enough that one process cannot open, cache, and scan every shard efficiently. With one storage shard, groups only change resource names. Small deployments can set spec.reader.readGroups.enabled: false to keep a single reader StatefulSet whose pods open every shard.

Reader autoscaling

spec.reader.autoscaling creates one HorizontalPodAutoscaler per reader StatefulSet, one per group when groups are enabled. A hot shard's group therefore scales without scaling the others:

spec:
  reader:
    replicas: 2 # Used as minReplicas when autoscaling omits it.
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 8
      targetCPUUtilizationPercentage: 70 # Default when no target is set.
      # targetMemoryUtilizationPercentage: 80
      # behavior: {} # autoscaling/v2 scale-up and scale-down policies.
    podTemplate:
      spec:
        containers:
          - name: metrics
            resources:
              requests: { cpu: "2", memory: 8Gi }

CPU is the default target because query load is CPU-bound once the cache is warm. A memory target is optional and is combined with the CPU target when both are set. Utilization is relative to the container's requests, so the reader container must request the targeted resources. The operator leaves the replica count of an autoscaled StatefulSet to its HorizontalPodAutoscaler. Every reader StatefulSet also gets a PodDisruptionBudget with maxUnavailable: 1.

The floor of the reader tier is S × minReplicas pods. Keep minReplicas at two or more where a shard must stay readable during node drains, because a group with one replica cannot serve its shard while that pod restarts.

Required benchmark matrix

Use the opt-in deployed scalability runner to generate Logs, Metrics, and Traces traffic. It is a standalone package and is not part of the standard regression suite.

Run each product for at least 30 minutes after warmup, then continue until compaction reaches steady state. Short tests that stop before compaction catches up overstate sustainable throughput.

For every workload, sweep load until one of the following occurs:

• Request latency or error rate violates the intended service objective.
• CPU remains above 80% or memory approaches its configured bound.
• SlateDB applies sustained write backpressure.
• L0 or compaction backlog grows without recovering.
• Object-store traffic or request throttling reaches its limit.
• Reader visibility lag exceeds the configured interval plus an allowed margin.

Record:

• Raw request bytes/s and decoded telemetry bytes/s.
• Entries/s, samples/s, or spans/s.
• Active streams, active series, or live traces and their churn.
• Batch-size and request-latency distributions.
• Logical bytes and operation count submitted to SlateDB.
• Object-store GET, PUT, and transferred bytes.
• CPU, RSS, cache hit rate, network, and local cache I/O.
• L0 count, compaction backlog, write stalls, and durability lag.
• Read latency under a representative concurrent query workload.

The minimum workload set is:

1. Direct, small client requests to expose request overhead.
2. Direct, 1–4 MiB client batches.
3. Buffer-coalesced 1–4 MiB per-shard writes when the buffer integration exists.
4. Default applied durability, with written as a throughput comparison.
5. durable mode as a latency-focused comparison.
6. Stable-cardinality and high-churn variants.
7. Concurrent representative reads to detect cache and object-store contention.
8. A scale-up across a routing-epoch cutover while running at 65% of validated steady-state capacity.

Publish both the saturation result and the recommended result after applying 35% operational headroom.

Scaling calculation

For a measured product and workload:

required writers =
    ceil(
        peak telemetry rate
        / measured per-writer saturation rate
        / 0.65
    )

When only byte measurements are available:

required writers =
    ceil(
        peak raw bytes/s
        * measured application write amplification
        / measured SlateDB logical bytes/s per writer
        / 0.65
    )

Use peak sustained rate rather than daily average. If an ingest buffer is deployed, writers may instead be sized to drain bursts within a stated backlog SLO:

drain time =
    buffered burst bytes
    / (aggregate writer drain bytes/s - continuing ingest bytes/s)

The denominator must remain positive. A buffer absorbs a burst; it does not make an under-provisioned steady-state write path sustainable.