Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
d4be2de
initial commit
acrylJonny Jan 21, 2025
660daf8
Delete grafana2
acrylJonny Jan 21, 2025
6cfbf5f
Merge branch 'master' into grafana-improvements
acrylJonny Jan 21, 2025
23e4eca
second commit
acrylJonny Jan 21, 2025
5468909
updating tests and adding tags and ownership options
acrylJonny Jan 21, 2025
e684eed
Merge branch 'master' into grafana-improvements
acrylJonny Jan 21, 2025
eda52c4
error updates and better formatting of docs
acrylJonny Jan 21, 2025
507c811
Update grafana_api.py
acrylJonny Jan 21, 2025
0e9f0dc
updating to use basemodel. Cleaning up code
acrylJonny Jan 22, 2025
c832e91
test updates
acrylJonny Jan 22, 2025
0fe68d2
Merge branch 'master' into grafana-improvements
acrylJonny Jan 22, 2025
c0e63d7
reorder libraries to be alphabetical
acrylJonny Jan 22, 2025
64a3bae
Update grafana_source.py
acrylJonny Jan 22, 2025
09b1ed8
adding unit tests
acrylJonny Jan 24, 2025
b06fd12
Merge branch 'master' into grafana-improvements
acrylJonny Jan 24, 2025
e0533ad
Adding optionals
acrylJonny Jan 24, 2025
27a9ceb
removing sqlparse requirement
acrylJonny Jan 24, 2025
a7e9e19
test updates
acrylJonny Jan 24, 2025
4714b24
Merge branch 'master' into grafana-improvements
acrylJonny Jan 24, 2025
f0f88c6
Merge branch 'master' into grafana-improvements
acrylJonny Jan 26, 2025
7646b89
addressing comments
acrylJonny Feb 14, 2025
6bd09ed
Merge branch 'master' into grafana-improvements
acrylJonny Feb 14, 2025
51310f9
addressing comments
acrylJonny Feb 14, 2025
54d142f
Improving overall experience
acrylJonny Feb 17, 2025
b4b7409
Merge branch 'master' into grafana-improvements
acrylJonny Feb 17, 2025
b4da306
Update constants.ts
acrylJonny Feb 17, 2025
40b75b8
Merge branch 'grafana-improvements' of https://github.com/acrylJonny/…
acrylJonny Feb 17, 2025
9bd5e7d
Merge branch 'master' into grafana-improvements
acrylJonny Feb 18, 2025
1dbe6f9
Update grafana_pre.md
acrylJonny Feb 18, 2025
8e150cd
test file updates
acrylJonny Feb 18, 2025
addfd15
Merge branch 'master' into grafana-improvements
acrylJonny Feb 18, 2025
40c2ff9
Merge branch 'master' into grafana-improvements
acrylJonny Feb 19, 2025
ac2e4cd
Merge branch 'master' into grafana-improvements
acrylJonny Feb 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
initial commit
acrylJonny committed Jan 21, 2025
commit d4be2de47e71f003fce608302cc96da3489dd9a3
1,379 changes: 1,379 additions & 0 deletions grafana2

Large diffs are not rendered by default.

1,188 changes: 1,117 additions & 71 deletions metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Thinking on the future troubleshooting, adding more instrumentation may help (some metrics in the report or info/debug log traces)

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,295 @@
{
"id": null,
"uid": "default",
"title": "Test Integration Dashboard",
"tags": ["test-tag", "integration-test"],
"timezone": "browser",
"schemaVersion": 36,
"version": 0,
"panels": [
{
"id": 1,
"type": "text",
"title": "Dashboard Information",
"gridPos": {
"x": 0,
"y": 0,
"w": 24,
"h": 3
},
"options": {
"content": "# Test Integration Dashboard\nThis dashboard contains test panels for DataHub integration testing with both PostgreSQL metrics and Prometheus system metrics.",
"mode": "markdown"
}
},
{
"id": 2,
"type": "timeseries",
"title": "Response Times by Dimension",
"description": "Response times tracked across different dimensions",
"datasource": {
"type": "postgres",
"uid": "test-postgres"
},
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"spanNulls": false
},
"color": {
"mode": "palette-classic"
},
"unit": "ms"
}
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 3
},
"targets": [
{
"datasource": {
"type": "postgres",
"uid": "test-postgres"
},
"editorMode": "code",
"format": "time_series",
"rawQuery": true,
"rawSql": "SELECT time, value, dimension FROM test_metrics WHERE metric = 'response_time' AND time > NOW() - interval '1 hour' ORDER BY time ASC;",
"refId": "A",
"sql": {
"columns": [
{
"parameters": [],
"type": "time"
},
{
"parameters": [
{
"name": "value",
"type": "column"
}
],
"type": "value"
}
]
}
}
]
},
{
"id": 4,
"type": "table",
"title": "Recent Metrics",
"description": "Recent metrics from all sources",
"datasource": {
"type": "postgres",
"uid": "test-postgres"
},
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"displayMode": "auto"
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "value"
},
"properties": [
{
"id": "custom.width",
"value": 150
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 11
},
"targets": [
{
"datasource": {
"type": "postgres",
"uid": "test-postgres"
},
"editorMode": "code",
"format": "table",
"rawQuery": true,
"rawSql": "SELECT time, metric, value, dimension FROM test_metrics WHERE time > NOW() - interval '1 hour' ORDER BY time DESC LIMIT 10;",
"refId": "A",
"sql": {
"columns": [
{
"parameters": [],
"type": "time"
},
{
"parameters": [
{
"name": "metric",
"type": "column"
}
],
"type": "string"
},
{
"parameters": [
{
"name": "value",
"type": "column"
}
],
"type": "number"
},
{
"parameters": [
{
"name": "dimension",
"type": "column"
}
],
"type": "string"
}
]
}
}
]
},
{
"id": 5,
"type": "stat",
"title": "Total Metrics Count",
"description": "Total number of metrics collected",
"datasource": {
"type": "postgres",
"uid": "test-postgres"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
}
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 3
},
"targets": [
{
"datasource": {
"type": "postgres",
"uid": "test-postgres"
},
"editorMode": "code",
"format": "table",
"rawQuery": true,
"rawSql": "SELECT COUNT(*) as count FROM test_metrics WHERE time > NOW() - interval '1 hour';",
"refId": "A",
"sql": {
"columns": [
{
"parameters": [
{
"name": "count",
"type": "column"
}
],
"type": "number"
}
]
}
}
]
},
{
"id": 6,
"type": "timeseries",
"title": "System Metrics",
"description": "Prometheus system metrics",
"datasource": {
"type": "prometheus",
"uid": "test-prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 10
}
}
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 19
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "test-prometheus"
},
"editorMode": "code",
"expr": "rate(process_cpu_seconds_total[5m])",
"legendFormat": "CPU Usage",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "test-prometheus"
},
"editorMode": "code",
"expr": "go_memstats_alloc_bytes",
"legendFormat": "Memory Usage",
"range": true,
"refId": "B"
}
]
}
],
"refresh": "5s",
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h"
]
},
"timezone": "browser",
"description": "A comprehensive test dashboard for integration testing with various panel types and data sources"
}

This file was deleted.

40 changes: 37 additions & 3 deletions metadata-ingestion/tests/integration/grafana/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -10,22 +10,56 @@ services:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_SECURITY_ADMIN_USER=admin
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
- GF_AUTH_DISABLE_LOGIN_FORM=false
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
- GF_FEATURE_TOGGLES_ENABLE=publicDashboards
volumes:
- grafana-storage:/var/lib/grafana
- ./provisioning:/etc/grafana/provisioning
- ./default-dashboard.json:/var/lib/grafana/dashboards/default-dashboard.json
- ./dashboards:/var/lib/grafana/dashboards
depends_on:
- postgres
postgres:
condition: service_healthy
prometheus:
condition: service_started
networks:
- grafana-network

postgres:
image: postgres:13
container_name: grafana-postgres
container_name: postgres
environment:
POSTGRES_DB: grafana
POSTGRES_USER: grafana
POSTGRES_PASSWORD: grafana
healthcheck:
test: [ "CMD-SHELL", "pg_isready -U grafana -d grafana" ]
interval: 10s
timeout: 5s
retries: 5
volumes:
- postgres-storage:/var/lib/postgresql/data
- ./postgres-init:/docker-entrypoint-initdb.d
networks:
- grafana-network

prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
networks:
- grafana-network

networks:
grafana-network:
driver: bridge

volumes:
grafana-storage:
Loading