Skip to content

linera-storage: add in-memory caches for immutable DB reads#6389

Draft
ndr-ds wants to merge 1 commit into
mainfrom
ndr-ds/dbstorage-immutable-caches
Draft

linera-storage: add in-memory caches for immutable DB reads#6389
ndr-ds wants to merge 1 commit into
mainfrom
ndr-ds/dbstorage-immutable-caches

Conversation

@ndr-ds

@ndr-ds ndr-ds commented May 26, 2026

Copy link
Copy Markdown
Contributor

Motivation

DbStorage pays a full Scylla/RocksDB round-trip on every call to four read methods that return permanently immutable data. On busy validators this contributes directly to Scylla read pressure (one node sustained ~7-8s p99 read spikes, with cache storms making it worse).

The four uncached paths:

  • read_certificate_hashes_by_heights(ChainId, BlockHeight) → CryptoHash
  • read_event_block_heightsEventId → BlockHeight
  • read_events_from_index — per-event bytes (keys still require a DB scan, but values now hit the cache)
  • read_network_description — single NetworkDescription value

All four are permanently immutable once written. They are safe to cache indefinitely.

Proposal

Add three new caches to StorageCaches:

  • block_hash_by_height: Arc<ValueCache<(ChainId, BlockHeight), CryptoHash>> — S3-FIFO bounded, consistent with the existing five caches
  • event_block_height: Arc<ValueCache<EventId, BlockHeight>> — same
  • network_description: Arc<OnceLock<NetworkDescription>> — single-value, zero overhead after first read

Read path: check-then-populate pattern (same as read_certificates_raw). Misses batch to DB; hits short-circuit. Write path: caches are populated in write_blobs_and_certificate at write time so write-then-read (the common path) is immediately cached.

read_events_from_index still scans DB for the key list (unavoidable), but now checks the existing event cache before bulk-reading values.

New CLI flags with conservative defaults (1 000 entries), overridden to larger values in the validator values.yaml:

  • --block-hash-by-height-cache-size (validator: 100 000)
  • --event-block-height-cache-size (validator: 50 000)

New Prometheus counters: READ_BLOCK_HASH_BY_HEIGHT_COUNTER, READ_EVENT_BLOCK_HEIGHT_COUNTER (labeled [CACHE]/[DB]). read_network_description gets [CACHE]/[DB] on the existing READ_NETWORK_DESCRIPTION_COUNTER.

Test Plan

  • cargo clippy clean on --all-targets --all-features and --no-default-features
  • Covered by existing integration tests exercising read_certificate_hashes_by_heights and read_events_from_index for the full round-trip

Release Plan

  • These changes should be backported to the latest testnet branch, then be released in a validator hotfix.

Links

@ndr-ds ndr-ds force-pushed the ndr-ds/dbstorage-immutable-caches branch from 27b6bfe to a02af66 Compare June 9, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant