Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add $lttb and $lttbMs implementation #720

Merged
merged 8 commits into from
Mar 7, 2025
Merged

add $lttb and $lttbMs implementation #720

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
fcf174f
add $lttb and $lttbMs macros to allow show down-sampled time series w…
Slach Feb 5, 2025
115ccf8
merge master into add_rttb
Slach Feb 5, 2025
2e4aeb9
fix eval_query, todo need interpret X-ClickHouse-Timezone response he…
Slach Feb 5, 2025
8c25185
Merge branch 'v4' of github.com:Altinity/clickhouse-grafana into add_…
Slach Feb 5, 2025
cc87925
fix failed tests
Slach Feb 5, 2025
99aa604
Merge branch 'v4' of github.com:Altinity/clickhouse-grafana into add_…
Slach Feb 10, 2025
c98ddd5
Merge branch 'v4' of github.com:Altinity/clickhouse-grafana into add_…
Slach Feb 24, 2025
1f68faf
Merge branch 'refs/heads/v4' into add_rttb
lunaticusgreen Mar 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# 3.4.0 (not released yet)
## Enhancements:
* add $lttb and $lttbMs macros to allow show down-sampled time series which will contains more outliers than avg or other kind of aggregation, fix https://github.com/Altinity/clickhouse-grafana/issues/500

# 3.3.1 (2024-12-27)
## Enhancements:
* Add using window functions instead of `runningDifference` and `neighbor` for macros, to avoid `allow_deprecated_error_prone_window_functions`, fix https://github.com/Altinity/clickhouse-grafana/issues/572
Expand Down
43 changes: 43 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,49 @@ GROUP BY t
ORDER BY t
```

---
### $lttb(buckets_number, [field1, ... fieldN], x_field, y_field) - allow show down-sampled time series which will contains more outliers than avg or other kind of aggregation

If bucket_number is `auto`, then it will calculated as `toUInt64( ($to-$from) / $interval )`
Example usage:

```sql
$lttb(auto, category, event_time, count(*) c)
FROM requests GROUP BY category
```

Query will be transformed into:

```sql
SELECT category, lttb_result.1 AS event_time, lttb_result.2 AS c FROM (
SELECT category, untuple(arrayJoin(lttb(toUInt64( ($to - $from) / $interval ))(event_time, cont(*) AS c))) AS lttb_result
FROM requests WHERE $timeFilter GROUP BY category
) ORDER BY event_time
```

---

---
### $lttbMs(buckets_number, [field1,... fieldN], x_field, y_field) - same as $lttb but for time series with ms

If bucket_number is `auto`, then it will calculated as `toUInt64( ($__to-$__from) / $__interval_ms )`

Example usage:

```sql
$lttbMs(100, event_time, count(*) c)
FROM requests
```

Query will be transformed into:

```sql
SELECT lttb_result.1 AS event_time, lttb_result.2 AS c FROM (
SELECT untuple(arrayJoin(lttb(100)(event_time, count(*) AS c))) AS lttb_result
FROM requests WHERE $timeFilterMs
) ORDER BY event_time
```

---

### $rateColumns(key, value) - is a combination of $columns and $rate
Expand Down
16 changes: 16 additions & 0 deletions docker/clickhouse/init_schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,3 +332,19 @@ SELECT
toUInt64(now() - INTERVAL number SECOND)*1000000000 + toUInt64(randUniform(0, 1000000000)) AS tUInt64_9,
number AS value
FROM numbers(86400);

DROP TABLE IF EXISTS default.test_lttb SYNC;
CREATE TABLE default.test_lttb(
event_time DateTime,
event_time_ms DateTime64(3),
category LowCardinality(String),
requests UInt64
) ENGINE=MergeTree() ORDER BY event_time;

INSERT INTO default.test_lttb
SELECT
now() - INTERVAL number SECOND,
now() - INTERVAL number SECOND + INTERVAL randUniform(0, 100) MILLISECOND,
'category' || toString(number%25),
rand() % 1000000
FROM numbers(86400);
183 changes: 183 additions & 0 deletions docker/grafana/dashboards/issue_500_lttb_and_lttbMs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "reproduce dashboards for https://github.com/Altinity/clickhouse-grafana/issues/500",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 54,
"links": [],
"panels": [
{
"datasource": {
"type": "vertamedia-clickhouse-datasource",
"uid": "P7E099F39B84EA795"
},
"description": "https://github.com/Altinity/clickhouse-grafana/issues/500",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.5.1",
"targets": [
{
"adHocFilters": [],
"adHocValuesQuery": "",
"add_metadata": true,
"contextWindowSize": "10",
"database": "default",
"datasource": {
"type": "vertamedia-clickhouse-datasource",
"uid": "P7E099F39B84EA795"
},
"dateTimeColDataType": "event_time",
"dateTimeType": "DATETIME",
"editorMode": "sql",
"extrapolate": true,
"format": "time_series",
"formattedQuery": "SELECT $timeSeries as t, count() FROM $table WHERE $timeFilter GROUP BY t ORDER BY t",
"interval": "",
"intervalFactor": 1,
"query": "$lttb(auto, category, event_time, count()) FROM $table WHERE $timeFilter GROUP BY category",
"refId": "A",
"round": "0s",
"showFormattedSQL": true,
"skip_comments": true,
"table": "test_lttb",
"useWindowFuncForMacros": true
},
{
"adHocFilters": [],
"adHocValuesQuery": "",
"add_metadata": true,
"contextWindowSize": "10",
"database": "default",
"datasource": {
"type": "vertamedia-clickhouse-datasource",
"uid": "P7E099F39B84EA795"
},
"dateTimeColDataType": "event_time",
"dateTimeType": "DATETIME",
"editorMode": "sql",
"extrapolate": true,
"format": "time_series",
"formattedQuery": "SELECT $timeSeries as t, count() FROM $table WHERE $timeFilter GROUP BY t ORDER BY t",
"hide": false,
"interval": "",
"intervalFactor": 1,
"query": "$lttbMs(auto, category, event_time, count()) FROM $table WHERE $timeFilter GROUP BY category",
"refId": "B",
"round": "0s",
"showFormattedSQL": true,
"skip_comments": true,
"table": "test_lttb",
"useWindowFuncForMacros": true
}
],
"title": "$lttb \\ $lttbMs auto buckets",
"type": "timeseries"
}
],
"preload": false,
"refresh": "",
"schemaVersion": 40,
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "$lttb and $lttbMs",
"uid": "lttb_lttb_ms",
"version": 2,
"weekStart": ""
}
70 changes: 37 additions & 33 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.23
require (
github.com/andybalholm/brotli v1.1.1
github.com/dlclark/regexp2 v1.11.4
github.com/grafana/grafana-plugin-sdk-go v0.260.1
github.com/grafana/grafana-plugin-sdk-go v0.263.0
github.com/hyperjumptech/jiffy v1.0.0
github.com/klauspost/compress v1.17.11
github.com/stretchr/testify v1.10.0
Expand All @@ -15,58 +15,61 @@ require (
require (
github.com/BurntSushi/toml v1.4.0 // indirect
github.com/antlr/antlr4 v0.0.0-20200124162019-2d7f727a00b7 // indirect
github.com/apache/arrow-go/v18 v18.1.0 // indirect
github.com/apache/arrow/go/v15 v15.0.2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cheekybits/genny v1.0.0 // indirect
github.com/chromedp/cdproto v0.0.0-20241110205750-a72e6703cd9b // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
github.com/chromedp/cdproto v0.0.0-20250203011601-a3c71a042730 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.6 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/elazarl/goproxy v0.0.0-20240909085733-6741dbfc16a1 // indirect
github.com/elazarl/goproxy v1.7.0 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/getkin/kin-openapi v0.128.0 // indirect
github.com/getkin/kin-openapi v0.129.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/google/flatbuffers v25.1.24+incompatible // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/grafana/otel-profiling-go v0.5.1 // indirect
github.com/grafana/pyroscope-go/godeltaprof v0.1.8 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus v1.0.1 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-plugin v1.6.2 // indirect
github.com/hashicorp/go-plugin v1.6.3 // indirect
github.com/hashicorp/yamux v0.1.2 // indirect
github.com/invopop/yaml v0.3.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/magefile/mage v1.15.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mailru/easyjson v0.9.0 // indirect
github.com/mattetti/filebuffer v1.0.1 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/oasdiff/yaml v0.0.0-20241214135536-5f7845c759c8 // indirect
github.com/oasdiff/yaml3 v0.0.0-20241214160948-977117996672 // indirect
github.com/oklog/run v1.1.0 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.20.5 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.60.1 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
Expand All @@ -75,28 +78,29 @@ require (
github.com/unknwon/log v0.0.0-20200308114134-929b1006e34a // indirect
github.com/urfave/cli v1.22.16 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.57.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.57.0 // indirect
go.opentelemetry.io/contrib/propagators/jaeger v1.32.0 // indirect
go.opentelemetry.io/contrib/samplers/jaegerremote v0.26.0 // indirect
go.opentelemetry.io/otel v1.32.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.32.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.32.0 // indirect
go.opentelemetry.io/otel/metric v1.32.0 // indirect
go.opentelemetry.io/otel/sdk v1.32.0 // indirect
go.opentelemetry.io/otel/trace v1.32.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
golang.org/x/exp v0.0.0-20241108190413-2d47ceb2692f // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.59.0 // indirect
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0 // indirect
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0 // indirect
go.opentelemetry.io/otel v1.34.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 // indirect
go.opentelemetry.io/otel/metric v1.34.0 // indirect
go.opentelemetry.io/otel/sdk v1.34.0 // indirect
go.opentelemetry.io/otel/trace v1.34.0 // indirect
go.opentelemetry.io/proto/otlp v1.5.0 // indirect
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.33.0 // indirect
golang.org/x/sys v0.28.0 // indirect
golang.org/x/net v0.34.0 // indirect
golang.org/x/sys v0.29.0 // indirect
golang.org/x/text v0.21.0 // indirect
golang.org/x/tools v0.27.0 // indirect
golang.org/x/tools v0.29.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/grpc v1.68.0 // indirect
google.golang.org/protobuf v1.35.2 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250127172529-29210b9bc287 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250127172529-29210b9bc287 // indirect
google.golang.org/grpc v1.70.0 // indirect
google.golang.org/protobuf v1.36.4 // indirect
gopkg.in/fsnotify/fsnotify.v1 v1.4.7 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading