What happens
On v0.130.1 (single-node ClickHouse, Docker compose), query_range for metrics intermittently returns existing data as empty for aggregation queries, and repeated identical queries return progressively more points — a cold-cache warm-up.
Metrics are scraped at 5s and the data is continuously present in ClickHouse (verified: samples_v4 median & p99 inter-sample gap = 5s over the last hour). But POST /api/v4/query_range for max(m) / min(m) / max(m)-min(m) over a short window returns far fewer points than exist, and grows on each identical repeat:
cache on : 4, 9, 14, 19, 24, 29 (same 3-min window, back-to-back; should be ~36 at 5s)
noCache : 4, 4, 4, 4, 4, 4 (consistent, but heavily downsampled)
Why it matters
- Dashboards show gaps / different data on each refresh.
- Alerts flap: a rule evaluates once per cycle and hits a cold-cache empty for its window; if the query uses
... or vector(<sentinel>) (to keep firing through a real outage), that empty becomes a false fire, and the next cycle warms and resolves → "resolved in 30s" flapping.
- Formula queries silently lose data:
sum(rate(a[2m]) + rate(b[2m]) + ...) returns empty when a component series is 0/absent (the un-summed rates fail to vector-match), whereas sum(rate(a[2m])) + sum(rate(b[2m])) + ... works. Any alert/panel built on the first form is silently disabled.
Reproduce
- Scrape any gauge at 5s into SigNoz.
POST /api/v4/query_range — queryType: promql, query: max(the_gauge), a 3-minute window, step: 5.- Repeat the identical request ~6× and count
series[0].values each time → it grows.
SELECT count(), quantile(0.5)(gap) FROM (... lagInFrame over samples_v4 ...) → the underlying data is a continuous 5s.
Environment
- SigNoz v0.130.1, single-node ClickHouse 25.5.6, Docker compose.
- Reproduced identically on a separate clean v0.130.1 instance, so it is not data/scrape specific.
Likely related: #5007, #9356, #9800.
Question
Is this the metrics query cache (CacheTTL / flux interval) cold-start, or a downsampling / step-interval issue? Is there a config — or a version where it's fixed — that makes query_range return the full stored resolution deterministically?
What happens
On v0.130.1 (single-node ClickHouse, Docker compose),
query_rangefor metrics intermittently returns existing data as empty for aggregation queries, and repeated identical queries return progressively more points — a cold-cache warm-up.Metrics are scraped at 5s and the data is continuously present in ClickHouse (verified:
samples_v4median & p99 inter-sample gap = 5s over the last hour). ButPOST /api/v4/query_rangeformax(m)/min(m)/max(m)-min(m)over a short window returns far fewer points than exist, and grows on each identical repeat:Why it matters
... or vector(<sentinel>)(to keep firing through a real outage), that empty becomes a false fire, and the next cycle warms and resolves → "resolved in 30s" flapping.sum(rate(a[2m]) + rate(b[2m]) + ...)returns empty when a component series is 0/absent (the un-summed rates fail to vector-match), whereassum(rate(a[2m])) + sum(rate(b[2m])) + ...works. Any alert/panel built on the first form is silently disabled.Reproduce
POST /api/v4/query_range—queryType: promql,query: max(the_gauge), a 3-minute window,step: 5.series[0].valueseach time → it grows.SELECT count(), quantile(0.5)(gap) FROM (... lagInFrame over samples_v4 ...)→ the underlying data is a continuous 5s.Environment
Likely related: #5007, #9356, #9800.
Question
Is this the metrics query cache (CacheTTL / flux interval) cold-start, or a downsampling / step-interval issue? Is there a config — or a version where it's fixed — that makes
query_rangereturn the full stored resolution deterministically?