Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 71 additions & 15 deletions ctats/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ _noun_

OTEL metrics, despite being an invaluable addition to service telemetry,
require an obnoxiously verbose setup and implementation. Ctats isn't
here to provide any new features. Instead in wants to make the current
here to provide any new features. Instead it wants to make the current
features more accessible and less painful.

### Step 1: Init OTEL with Clues
Expand All @@ -36,7 +36,7 @@ func main() {
```go
func main() {
// ...
ctx, err = ctats.Initialize(ctx)
ctx, err = ctats.Initialize(ctx)
// ...
}
```
Expand All @@ -46,11 +46,11 @@ func main() {
```go
func main() {
// We're not kidding, this step is purely optional.
ctx, err := ctats.RegisterHistogram(
ctx, err := ctats.RegisterSum(
ctx,
"http.server.latency", // Name
"ms", // Unit
"New user additions.", // Description
"http.server.requests", // Name
"1", // Unit
"Incoming HTTP requests by status code.", // Description
)
}
```
Expand All @@ -59,10 +59,12 @@ func main() {

```go
func handler(ctx context.Context) {
//...
ctats.Histogram[int64]("http.server.latency").Record(latency)
//...

// ...
ctats.Sum[int64]("http.server.requests").
With("status_code", statusCode).
Inc(ctx)
// ...
}
```

## How it works
Expand Down Expand Up @@ -97,20 +99,72 @@ values are `float64`s behind the scenes. Easier to avoid the problem
of potential conflicts altogether. What, would you prefer that we
panic?

## Corner Case: histogram bucket definitions
## Histograms

Comment thread
ryanfkeepers marked this conversation as resolved.
Histograms can be a bit more work than the other types because you have to think
about your data's distribution ahead of time. Sure, you can run with whatever
OTEL uses as the default (15 buckets, scaling exponentially up to 10000), but is that
really the best showcase for your data? Probably not.

In case you're new to all this business,
[here's a good read](https://signoz.io/blog/opentelemetry-histogram/)
to catch you up with how histograms work under the hood.

You can't define your histogram buckets with Ctats. Why? Because
[OTEL doesn't let you define them at runtime either](https://github.com/open-telemetry/opentelemetry-go/issues/3826).
Comment thread
robertschonfeld marked this conversation as resolved.
You'll have to take it up with the package authors, not us.
So, how do you set yourself up for histogram success using ctats?
Just register your buckets on init! Simple as that.

## Sum vs Counter vs Gauge
```go
func main() {
boundaries := ctats.MakeExponentialHistogramBoundaries(1, 60_000, 15, 1)
ctx, err := ctats.RegisterHistogram(
ctx,
"op.latency",
"ms",
"End-to-end operation latency.",
ctats.WithBoundaries(boundaries...),
)
}

func handler(ctx context.Context) {
ctats.Histogram[int64]("op.latency").Record(ctx, elapsed)
}
```

Registering is optional. You can also pass `WithBoundaries` directly to the
factory and the instrument is created on the first `Record` call. Just keep
in mind that the first creation wins — if the same id was already registered
or recorded against with different boundaries, the new ones are silently
ignored.

### Picking your boundaries

Use `MakeExponentialHistogramBoundaries` to generate logarithmically-spaced
buckets. `low` and `high` determine the supported range of your metric.

The optional `scalingFactor` warps the bucket distribution. At 1 you get
uniform log-spacing. Above 1, more edges cluster near `low` (useful for
latency, where data tends to clump at the low end). Between 0 and 1, more
edges cluster near `high`. Values ≤ 0 default to 1.

```go
// example: measuring http server latencies in ms up to 60s
boundaries := ctats.MakeExponentialHistogramBoundaries(1, 60_000, 15, 1)

ctats.Histogram[int64](
"job.duration",
ctats.WithBoundaries(boundaries...),
).Record(ctx, elapsed)
```
Comment thread
robertschonfeld marked this conversation as resolved.

## Which metric type should I use?

Feeling overwhelmed? Not sure which type to pick? Just answer
these simple questions and you'll be a master in no time!

* Sum -> OTEL Counter
* Counter -> OTEL UpDownCounter
* Gauge -> OTEL Gauge (who knew?)
* Histogram -> OTEL Histogram (surprise!)

Do you need `Delta Temporality`? Use a Sum, it's your only option!

Expand All @@ -119,6 +173,8 @@ Do you need to decrement values? Use a Counter!
Do you need have a single threaded, single source of truth? Try
a Gauge!

Do you need statistics such as percentiles? Use a Histogram!

Sums are the most foolproof option all around. Plug one in,
count away. Counters are nearly as good, if it weren't for the
temporality constraint. For monotonically increasing values,
Expand Down
121 changes: 108 additions & 13 deletions ctats/histogram.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package ctats
import (
"context"
"log"
"math"

"github.com/pkg/errors"
"go.opentelemetry.io/otel/metric"
Expand All @@ -11,12 +12,89 @@ import (
"github.com/alcionai/clues/internal/node"
)

// MakeExponentialHistogramBoundaries returns count boundaries spaced logarithmically
// between low and high (both inclusive). For background on explicit bucket histograms
// and how boundaries map to OTel buckets, see the OTel metrics SDK spec
// (navigate to the "Explicit Bucket Histogram Aggregation" section):
// https://opentelemetry.io/docs/specs/otel/metrics/sdk/
//
// scalingFactor warps the position distribution between low and high. At 1,
// positions are uniformly log-spaced — constant growth ratio. Values above 1
// pack more buckets toward low (useful when data clusters at the low end).
// Values between 0 and 1 pack more buckets toward high. Values ≤ 0 are invalid
// and default to 1.
//
// Example:
//
// MakeExponentialHistogramBoundaries(1, 60_000, 15, 1)
// // → [1 2 5 11 23 51 112 245 537 1179 2588 5679 12461 27344 60000]
//
// MakeExponentialHistogramBoundaries(10, 1000, 5, 0.5)
// // → [10 100 260 540 1000] (denser at high end)
//
// MakeExponentialHistogramBoundaries(10, 1000, 5, 1)
// // → [10 32 100 316 1000] (uniform log-spacing)
//
// MakeExponentialHistogramBoundaries(10, 1000, 5, 2)
// // → [10 13 32 133 1000] (denser at low end)
func MakeExponentialHistogramBoundaries(
low, high float64,
count int,
scalingFactor float64,
) []float64 {
if scalingFactor <= 0 {
scalingFactor = 1
Comment thread
robertschonfeld marked this conversation as resolved.
}

if count < 2 {
return []float64{low, high}
}

b := make([]float64, count)

for i := range b {
t := math.Pow(float64(i)/float64(count-1), scalingFactor)
b[i] = math.Round(low * math.Pow(high/low, t))
}

b[0] = low // guarantee exact floor, no rounding drift
b[count-1] = high // guarantee exact ceiling, no rounding drift

return b
}

type histogramCfg struct {
boundaries []float64
}

func (c histogramCfg) appendOpts(
opts ...metric.Float64HistogramOption,
) []metric.Float64HistogramOption {
if len(c.boundaries) > 0 {
opts = append(opts, metric.WithExplicitBucketBoundaries(c.boundaries...))
}

return opts
}

type HistogramOption func(*histogramCfg)

// WithBoundaries sets explicit bucket boundaries on the histogram.
// Boundaries are passed to the OTel SDK at instrument creation time and are
// ignored if a matching MeterProvider View is already configured.
func WithBoundaries(boundaries ...float64) HistogramOption {
return func(c *histogramCfg) {
c.boundaries = boundaries
}
}

// getOrCreateHistogram attempts to retrieve a histogram from the
// context with the given ID. If it is unable to find a histogram
// with that ID, a new histogram is generated.
func getOrCreateHistogram(
ctx context.Context,
id string,
cfg histogramCfg,
) (recorder, error) {
id = formatID(id)
b := fromCtx(ctx)
Expand All @@ -36,7 +114,10 @@ func getOrCreateHistogram(
return nil, cluerr.Stack(errNoNodeInCtx)
}

hist, err := nc.OTELMeter().Float64Histogram(id)
opts := cfg.appendOpts()

// register the histogram
hist, err := nc.OTELMeter().Float64Histogram(id, opts...)
if err != nil {
return nil, errors.Wrap(err, "making new histogram")
}
Expand All @@ -50,17 +131,19 @@ func getOrCreateHistogram(

// RegisterHistogram introduces a new histogram with the given unit and description.
// If RegisterHistogram is not called before updating a metric value, a histogram with
// no unit or description is created. If RegisterHistogram is called for an ID that
// no unit or description is created. If RegisterHistogram is called for an ID that
// has already been registered, it no-ops.
func RegisterHistogram(
ctx context.Context,
// all lowercase, period delimited id of the histogram. Ex: "http.response.status_code"
// all lowercase, period delimited id of the histogram. Ex: "http.response.size"
id string,
// (optional) the unit of measurement. Ex: "byte", "kB", "fnords"
unit string,
// (optional) a short description about the metric.
// Ex: "number of times we saw the fnords".
description string,
// (optional) histogram specific options
opts ...HistogramOption,
) (context.Context, error) {
id = formatID(id)

Expand All @@ -82,18 +165,24 @@ func RegisterHistogram(
return ctx, errors.New("no clues in ctx")
}

opts := []metric.Float64HistogramOption{}
var cfg histogramCfg
for _, o := range opts {
o(&cfg)
}

var metricHistogramOpts []metric.Float64HistogramOption

if len(description) > 0 {
opts = append(opts, metric.WithDescription(description))
metricHistogramOpts = append(metricHistogramOpts, metric.WithDescription(description))
}

if len(unit) > 0 {
opts = append(opts, metric.WithUnit(unit))
metricHistogramOpts = append(metricHistogramOpts, metric.WithUnit(unit))
}

// register the histogram
hist, err := nc.OTELMeter().Float64Histogram(id, opts...)
metricHistogramOpts = cfg.appendOpts(metricHistogramOpts...)

hist, err := nc.OTELMeter().Float64Histogram(id, metricHistogramOpts...)
if err != nil {
return ctx, errors.Wrap(err, "creating histogram")
}
Expand All @@ -107,17 +196,23 @@ func RegisterHistogram(
// If a Histogram instance has been registered for that ID, the
// registered instance will be used. If not, a new instance
// will get generated.
func Histogram[N number](id string) histogram[N] {
return histogram[N]{base: base{id: formatID(id)}}
func Histogram[N number](id string, opts ...HistogramOption) histogram[N] {
hgm := histogram[N]{base: base{id: formatID(id)}}
for _, o := range opts {
o(&hgm.histogramCfg)
}

return hgm
}

// histogram provides access to the factory functions.
type histogram[N number] struct {
base
histogramCfg
}

func (c histogram[N]) With(kvs ...any) histogram[N] {
return histogram[N]{base: c.with(kvs...)}
return histogram[N]{base: c.with(kvs...), histogramCfg: c.histogramCfg}
}

type recorder interface {
Expand All @@ -128,9 +223,9 @@ type noopRecorder struct{}

func (n noopRecorder) Record(context.Context, float64, ...metric.RecordOption) {}

// Add increments the histogram by n. n can be negative.
// Record records the measurement of n in the histogram.
Comment thread
robertschonfeld marked this conversation as resolved.
func (c histogram[number]) Record(ctx context.Context, n number) {
hist, err := getOrCreateHistogram(ctx, c.getID())
hist, err := getOrCreateHistogram(ctx, c.getID(), c.histogramCfg)
if err != nil {
log.Printf("err getting histogram: %+v\n", err)
return
Expand Down
Loading
Loading