Allow custom bucket boundaries per Histogram by robertschonfeld · Pull Request #300 · alcionai/clues

robertschonfeld · 2026-04-24T16:21:33Z

The OTel Go SDK uses explicit bucket boundaries that top out at 10,000. Any observation above that lands in the +Inf overflow bucket, and since Kibana's percentile() uses linear interpolation within buckets it silently maxes out at 10,000. Customizing the bucket boundaries is needed to measure latencies above 10,000

The OTel mechanism is explicit bucket boundaries via metric.WithExplicitBucketBoundaries at instrument creation time — per-instrument, not a global MeterProvider view.

Changes:

ExponentialBoundaries(min, max, count) — logarithmically-spaced buckets mirroring Prometheus's ExponentialBucketsRange
DefaultLatencyBoundariesMs — 20 buckets from 1–60,000
WithBoundaries(...) HistogramOption on Histogram[N] and RegisterHistogram
Tests covering the math, option propagation, and end-to-end Record bucket placement via a ManualReader-backed OTel context

ryanfkeepers

Please see comments. In particular:

unit test fixups.
automatic usage of the default boundaries.
removal of the histogramConfig struct.

ryanfkeepers · 2026-04-24T16:31:16Z

+// DefaultLatencyBoundariesMs are logarithmically-spaced bucket boundaries from
+// 1 to 60_000, suitable for measuring operation latency in milliseconds up to 60s.
+// Use with WithBoundaries to avoid the OTel SDK default ceiling of 10,000.
+var DefaultLatencyBoundariesMs = ExponentialBoundaries(1, 60_000, 20)


This value is only getting used in tests, and is not actually applied as the default. See other comments for usage suggestions.

It's not clear how the default boundaries in ctats should be optimized so I decided to keep the OTEL defaults. Making PresetLatencyBoundariesMs default could in theory worsen the precision for someone measuring smaller values. We still could make that change in the future though, especially if we get more experience with modifying the boundaries.

For best results, users must choose their boundaries for each metric based on the expected distribution of its values. After this change, I will update all of our call sites to do so. ctats provides PresetLatencyBoundariesMs and ExponentialBoundaries(min, max float64, count int) as utils for some reasonable boundaries.

Perhaps the naming was confusing so I renamed from DefaultLatencyBoundariesMs to PresetLatencyBoundariesMs.

Mmm, no, this is not an issue of wording. Any choice of names will produce the same problem. That is, I think you're overfitting to a known problem case in your environment. If we want clues to provide a set of presets, then we're saying that clues knows- and is authoritative about- the best possible histogram layouts for one or more standard scenarios. We could probably come up with a sufficient solution, sure. But at this time I don't see the benefit in taking on that authority.

For now I recommend that we drop this value. If you feel strongly about pursuing the idea further (which is also fine!) then we can do that in a follow-up pr,

Ok, removed. We will have this as a constant in our clients then.

ryanfkeepers · 2026-04-28T14:57:17Z

+
+// ExponentialBoundaries returns count boundaries spaced logarithmically between
+// min and max (both inclusive), mirroring Prometheus's ExponentialBucketsRange:
+// https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#ExponentialBucketsRange


We're not working with prometheus here. Are there any otel docs for the same info?

Linked to the otel doc about the Explicit Bucket Histogram Aggregation. That docs also refers to being inspired by prometheus which is the source for why we are using logarithmic spacing in the first place.

ryanfkeepers · 2026-04-28T15:05:19Z

+//
+//	ExponentialBoundaries(1, 60_000, 15)
+//	// → [1 2 5 11 23 51 112 245 537 1179 2588 5679 12461 27344 60000]
+func ExponentialBoundaries(min, max float64, count int) []float64 {


Two thoughts on this func, now that I've slept on it: first, let's verbify the name. Second, while I don't mind the default scaling, I think it would be appropriate to allow the caller to define their own scaling factor for further control. Any value less <= 1 should use the current default.

Suggested change

func ExponentialBoundaries(min, max float64, count int) []float64 {

func MakeExponentialHistogramBoundaries(min, max, factor float64, count int) []float64 {

Added scaling factor with the effect of skewing the boundaries towards the low end of the range. Is this what you had in mind?

Are you comfortable with this function or do we want to go deeper into the maths of what is optimal? I am satisfied with roughly following the logarithmic distribution of the otel default "inspired by prometheus".

Will be good to test this against app log based calculations which are exact.

I don't think we need to be too scientific. After all, this is just a quick way for someone to get an "approximately useful" set of buckets. They can always define their own if it needs to be exact according to some range.

Co-authored-by: Keepers <ryan.keepers@veeam.com>

ryanfkeepers

A couple remaining tiny nits. Thanks for all the effort!

ryanfkeepers · 2026-04-28T17:08:16Z

+//
+//	MakeExponentialHistogramBoundaries(10, 1000, 5, 2)
+//	// → [10 13 32 133 1000]    (denser at low end, same range)
+func MakeExponentialHistogramBoundaries(min, max float64, count int, scalingFactor float64) []float64 {


style nit, since this is getting to be a long line

Suggested change

func MakeExponentialHistogramBoundaries(min, max float64, count int, scalingFactor float64) []float64 {

func MakeExponentialHistogramBoundaries(

min, max float64,

count int,

scalingFactor float64,

) []float64 {

ryanfkeepers · 2026-04-28T17:08:43Z

+//	// → [10 13 32 133 1000]    (denser at low end, same range)
+func MakeExponentialHistogramBoundaries(min, max float64, count int, scalingFactor float64) []float64 {
+	if scalingFactor <= 1 {
+		scalingFactor = 1


is 1 correct? Should this use the old scaling factor evaluation?

(if so, we need a div-by-0 protection, too)

Suggested change

scalingFactor = 1

scalingFactor = math.Pow(max/min, 1/float64(count-1))

ryanfkeepers · 2026-04-28T17:10:04Z

+	boundaries []float64
+}
+
+func (c histogramCfg) appendOpts(opts []metric.Float64HistogramOption) []metric.Float64HistogramOption {


nit: cause variadics are just a little nicer for things like these.

Suggested change

func (c histogramCfg) appendOpts(opts []metric.Float64HistogramOption) []metric.Float64HistogramOption {

func (c histogramCfg) appendOpts(opts ...metric.Float64HistogramOption) []metric.Float64HistogramOption {

Allow custom bucket boundaries per Histogram

cc5a4f3

ryanfkeepers requested changes Apr 24, 2026

View reviewed changes

robertschonfeld added 7 commits April 28, 2026 13:17

histogram style

34f4e80

PresetLatencyBoundariesMs

7d8817e

PresetLatencyBoundariesMs use 15 buckets

58028a9

test boundaries

7bc6f05

TestHistogramFirstBoundariesWin

2b76533

doc

fcd2d89

comments

c501546

ryanfkeepers reviewed Apr 28, 2026

View reviewed changes

robertschonfeld and others added 10 commits April 28, 2026 17:49

Update ctats/README.md

75eeff2

Co-authored-by: Keepers <ryan.keepers@veeam.com>

Update ctats/README.md

dfd64bc

Co-authored-by: Keepers <ryan.keepers@veeam.com>

Update ctats/README.md

e8a41e4

Co-authored-by: Keepers <ryan.keepers@veeam.com>

Update ctats/README.md

10edad3

Co-authored-by: Keepers <ryan.keepers@veeam.com>

Update ctats/README.md

f0da24d

Co-authored-by: Keepers <ryan.keepers@veeam.com>

Update ctats/README.md

7358d84

Co-authored-by: Keepers <ryan.keepers@veeam.com>

scaling factor

2cfc1fd

otel link

e1250e8

appendOpts()

4385692

example

b3a0e80

ryanfkeepers approved these changes Apr 28, 2026

View reviewed changes

	func ExponentialBoundaries(min, max float64, count int) []float64 {
	func MakeExponentialHistogramBoundaries(min, max, factor float64, count int) []float64 {

	scalingFactor = 1
	scalingFactor = math.Pow(max/min, 1/float64(count-1))

	func (c histogramCfg) appendOpts(opts []metric.Float64HistogramOption) []metric.Float64HistogramOption {
	func (c histogramCfg) appendOpts(opts ...metric.Float64HistogramOption) []metric.Float64HistogramOption {

Conversation

robertschonfeld commented Apr 24, 2026

Uh oh!

ryanfkeepers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanfkeepers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants