Some data is missing from Prompp thanos sidecar

[alik-nikitin]

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Version

v.0.6.1

Expected Behavior

All fresh data from Prometheus appears as expected.

Actual Behavior

Some fresh data is missing when querying from thanos-query service. After 2 hours these data appers in thanos archive and we able to see them.

Prom++ with actual data existance

Thanos Query show some data is missing

Steps To Reproduce

Installed default thanos and kube-prometheus-stack with replaced image of Prom++.

Additional Information

Prom++, thanos-sidecar, thanos-query do not display any error messages on log.

Logs

[vporoshok]

It looks like thanos miss some historical block. Does the compactor have some errors or warnings in log?

[alik-nikitin]

No, compactor is working without any error/warning messages.
Also, this issue repeating on the second installation.

[vporoshok]

All data is missing? Since several hours any data is available through the Thanos?

[vporoshok]

sidecar also doesn't have any errors or warnings in logs?

[alik-nikitin]

This missing interval (~6:49 - ~7:24) is keep missing for 2 hours. After 2 hours the data is appearing in thanos. After some time the new interval is missing, for example 8:49-9:24.
Thanos-sidecar also doesn't have any error/warning messages, only succesfull 'msg="upload new block"'

[vporoshok]

And for missed intervals blocks successfully uploaded too?

Prompp write logs like:

ts=2025-11-01T04:27:34.755Z caller=compact.go:547 level=info component=tsdb msg="compact blocks" count=3 mint=1761962400001 maxt=1761969599977 ulid=01K8YTPZ7SJEEVN09D2BK1XPKV sources="[01K8YKVP97GN839XW1NKZVAQ0J 01K8YTPP79DRH784B71Y4Y5M38 01K8YTPTATXH4NHDD55J18RG10]" duration=14.569857315s

Can you check that all compaction results (01K8YTPZ7SJEEVN09D2BK1XPKV from logs example) are successfully uploaded to S3?

Another question: do you enable option --shipper.upload-compacted?

[alik-nikitin]

Prompp doesn't have logs like you sent.

ts=2025-11-01T02:50:22.972Z caller=receiver.go:832 level=info component=receiver pp_caller=storage.go:310 msg="QUERYABLE STORAGE: head { id: 2f450054-2584-447f-b052-d637db4863df, generation: 105 } persisted, closed and discarded"
ts=2025-11-01T04:43:29.605Z caller=receiver.go:832 level=info component=receiver pp_caller=storage.go:215 msg="QUERYABLE STORAGE: head { id: 51f887c0-f2ff-440c-a990-5f6f9666756a, generation: 106 } added"
ts=2025-11-01T04:45:22.807Z caller=receiver.go:832 level=info component=receiver pp_caller=storage.go:187 msg="QUERYABLE STORAGE: head { id: 51f887c0-f2ff-440c-a990-5f6f9666756a, generation: 106 } persisted, duration: 23.201940025s"
ts=2025-11-01T04:45:59.650Z caller=db.go:1833 level=info component=tsdb msg="Deleting obsolete block" block=01K8W9AJXPT56PW4CSC7DDBQ0W
ts=2025-11-01T04:45:59.672Z caller=db.go:1833 level=info component=tsdb msg="Deleting obsolete block" block=01K8W9AJXSNDKHJGW65KHJR1S7
ts=2025-11-01T04:45:59.702Z caller=db.go:1833 level=info component=tsdb msg="Deleting obsolete block" block=01K8W9AJXWMVB2ZDA2YBRPBHAD
ts=2025-11-01T04:45:59.728Z caller=db.go:1833 level=info component=tsdb msg="Deleting obsolete block" block=01K8W9AJXQ4411FQP3SX9N7ZTZ

option --shipper.upload-compacted disabled

[vporoshok]

What value is set for the storage.tsdb.retention.time option?

[vporoshok]

As I can see from the log messages, the blocks are deleted 37 seconds after being created. Could this cause them to not have enough time to be uploaded to S3?

[alik-nikitin]

storage.tsdb.retention.time=24h

S3 should not be a problem. Manual upload/download is fast.

[vporoshok]

I think that understand what lead to this problem. We will add feature-flag to disable internal prom++ compaction. Could you set custom env variable for prom++ in your installation?

[alik-nikitin]

Sure, let me know which variable should be added

[vporoshok]

Sorry for the delay. Please try updating to version 0.6.4 and enable the feature flag disable_block_compaction. For enabling feature flag use environment variable PROMPP_FEATURES see Dockerfile for example and default flags.

[alik-nikitin]

Thanks. I didn't find a convenient way to overwrite environment variables inside prometheus, so I've rebuilt new prompp:0.6.4 as

FROM prompp/prompp:0.6.4
ENV PROMPP_FEATURES="disable_block_compaction,unload_data_storage,head_copy_series_on_rotate,head_read_concurrency=1"

and launched it. But it didn't fixed issue.