-
Notifications
You must be signed in to change notification settings - Fork 308
feat(satp): added predefined grafana dashboards and alerts #4023
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
RafaelAPB
merged 1 commit into
hyperledger-cacti:main
from
JJSantos22:multi_chain_metrics
Oct 30, 2025
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # Grafana / OTEL-LGTM Provisioning Guide | ||
|
|
||
| This repository defines the provisioning setup for **Grafana** as part of the **[OpenTelemetry LGTM](https://hub.docker.com/r/grafana/otel-lgtm) (Loki, Grafana, Tempo, Prometheus)** stack. | ||
| Provisioning allows you to **automatically configure dashboards, alerting rules, and contact points** through YAML files — ensuring consistent, version-controlled observability configurations. | ||
|
|
||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| Provisioning in Grafana means preloading configuration files that are applied automatically when Grafana starts. | ||
| This setup covers three main provisioning categories: | ||
|
|
||
| 1. **Dashboards** – Predefined visualizations for metrics, logs, and traces. | ||
| 2. **Alerting Rules** – Conditions that trigger alerts based on metric or log thresholds. | ||
| 3. **Contact Points** – Notification destinations for alert delivery (e.g., email, Discord, Slack). | ||
|
|
||
| All provisioning files are placed inside the docker image under the directory `/otel-lgtm/grafana/conf/provisioning/` (or equivalent custom mount path defined in the [`docker-compose-satp.yml`](../docker-compose-satp.yml)). | ||
|
|
||
| --- | ||
|
|
||
| ## Directory Structure | ||
|
|
||
| To provision the monitor system with dashboards, alerts or contact points use the following repository layout: | ||
|
|
||
| provisioning/<br> | ||
| ├── dashboards/<br> | ||
| │ ├── [grafana-dashboards.yaml](#grafana-dashboardsyaml)     *# Dashboard provisioning configuration*<br> | ||
| │ ├── dashboard-#1.json       *# Dashboard #1 definition*<br> | ||
| │ ├── ...<br> | ||
| │ └── [dashboard-#n.json](#dashboard-njson)       *# Dashboard #n definition*<br> | ||
| ├── alerting/<br> | ||
| │ ├── alert-#1.yaml          *# Alert group and rule definitions #1*<br> | ||
| │ ├── ...<br> | ||
| │ ├── [alert-#m.yaml](#alert-myaml)         *# Alert group and rule definitions #m*<br> | ||
| │ ├── contact-point-#1.yaml      *# Contact point definitions #1*<br> | ||
| │ ├── ...<br> | ||
| │ └── [contact-point-#p.yaml](#contact-point-pyaml)      *# Contact point definitions #p*<br> | ||
|
|
||
| ### grafana-dashboards.yaml | ||
|
|
||
| This file contains the list of dashboards to be made available on start of the docker image. Each entry in the providers list defines a new dashboard. | ||
|
|
||
| ### dashboard-#n.json | ||
|
|
||
| An example of a dashboard, that must be included in the [grafana-dashboards.yaml](#grafana-dashboardsyaml). Each dashboard is defined in a separate JSON file | ||
|
|
||
| Official documentation for dashboard provisioning available [here](https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards). | ||
|
|
||
| ### alert-#m.yaml | ||
|
|
||
| An example of an alert. Alerts are defined in yaml files. | ||
|
|
||
| Official documentation for alert provisioning available [here](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/#import-alert-rules). | ||
|
|
||
| ### contact-point-#p.yaml | ||
|
|
||
| An example of a contact point. Contact points are defined in yaml files. | ||
|
|
||
| Official documentation for contact point provisioning available [here](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/#import-contact-points). | ||
|
|
||
| ## Usage Explanation | ||
|
|
||
| To customize the available dashboards, alerts and contact points, there are some files that require creation or modification. To create these elements there are 2 possible options: | ||
|
|
||
| - Read the [official documentation](https://grafana.com/docs/grafana/latest/) (less intuitive). | ||
| - Initiate the docker image, create the element and export it (more interactive). | ||
|
|
||
| In this explanation, we will provide a step-by-step tutorial on how to create a [dashboard](#dashboard-creation), an [alert rule](#alert-rule-creation) or a [contact point](#contact-point-creation) with the help of Grafana's built-in tools (the second option). | ||
|
|
||
| ### Dashboard Creation | ||
|
|
||
| 1. Go to the [`test file`](../src/test/typescript/integration/monitoring/functionality.test.ts) for monitoring and comment the lines that call the function `stopDockerComposeService`. | ||
|
|
||
| 2. Run the test using `npx jest ./packages/cactus-plugin-satp-hermes/src/test/typescript/integration/monitoring/functionality.test.ts` from the project root. | ||
|
|
||
| *Note: This is done to make the metrics appear in Grafana, facilitating the process of creation.* | ||
|
|
||
| 3. Login into Grafana using the following [link](http://localhost:3000/login) using `admin` as both *username* and *password*. | ||
|
|
||
| *Note: You might require to change the address in case it is not running on localhost.* | ||
|
|
||
| 4. Access the Grafana dashboard endpoint using the following [link](http://localhost:3000/dashboards). | ||
|
|
||
| *Note: You might require to change the address in case it is not running on localhost.* | ||
|
|
||
| 5. Click the `New` button and then the `New dashboard` button. | ||
|
|
||
| 6. Click the `Add visualization` and select a data source for the data visualization. | ||
|
|
||
| 7. Configure the panel with the desired data. | ||
|
|
||
| 8. Click `Save dashboard`, name the dashboard and click `Save`. | ||
|
|
||
| 9. If more panels are desired, click `Add` and then `Visualization`. Redo step 7 and 8. | ||
|
|
||
| 10. After all panels are added, click `Exit edit` and then, `Export` followed by `Export as code`. | ||
|
|
||
| 11. Click `Download file`, then move the file to the folder `grafana/provisioning/dashboards/` inside the package for the SATP-Hermes project. | ||
|
|
||
| 12. On the file [grafana-dashboards.yaml](./provisioning/dashboards/grafana-dashboards.yaml), create a new entry by copying an existing example one and change the path to have the name of the new dashboard and change the property name of the dashboard itself. | ||
|
|
||
| 13. Kill the running docker image for the container regarding otel-lgtm, rerun the test from step 2 and check if the dashboard is automatically provisioned. | ||
|
|
||
| 14. Go to the [`test file`](../src/test/typescript/integration/monitoring/functionality.test.ts) for monitoring and uncomment the commented lines that call the function `stopDockerComposeService`. | ||
|
|
||
| ### Contact Point Creation | ||
|
|
||
| *Note: This section appears before alert creation since alerts rely on the existence of established contact points.* | ||
|
|
||
| 1. Access the Grafana dashboard endpoint using the following [link](http://localhost:3000/alerting/notifications). | ||
|
|
||
| *Note: You might require to change the address in case it is not running on localhost.* | ||
|
|
||
| 2. Click the `Create contact point` button. | ||
|
|
||
| 3. Browse the integration options, selecting the one that better suits the use case (eg. discord, email, etc.). | ||
|
|
||
| 4. Fill the name of the contact point and the specific details of the integration option. | ||
|
|
||
| 5. Click `Test` to assess the correct functioning of the contact point. | ||
|
|
||
| 6. Click `Save contact point`. | ||
|
|
||
| 7. Click `More` on the newly created contact point, followed by `Export`. | ||
|
|
||
| 8. Click `Download`, then move the file to the folder `grafana/provisioning/alerting/` inside the package for the SATP-Hermes project. | ||
|
|
||
| ### Alert Rule Creation | ||
|
|
||
| 1. Go to the [`test file`](../src/test/typescript/integration/monitoring/functionality.test.ts) for monitoring and comment the lines that call the function `stopDockerComposeService`. | ||
|
|
||
| 2. Run the test using `npx jest ./packages/cactus-plugin-satp-hermes/src/test/typescript/integration/monitoring/functionality.test.ts` from the project root. | ||
|
|
||
| *Note: This is done to make the metrics appear in Grafana, facilitating the process of creation.* | ||
|
|
||
| 3. Access the Grafana dashboard endpoint using the following [link](http://localhost:3000/alerting/list). | ||
|
|
||
| *Note: You might require to change the address in case it is not running on localhost.* | ||
|
|
||
| 4. Click the `New alert rule` button. | ||
|
|
||
| 5. Fill in the name and select the metric to track. | ||
|
|
||
| 6. Configure the threshold that should trigger the alarm. | ||
|
|
||
| 7. Select the folder for your rule or click `New folder` to create a new one, giving it a name and clicking `Create`. | ||
|
|
||
| 8. Select the evaluation group for your rule (periodicity of evaluation) or click `New evaluation group` to create a new one, giving it a name and an evaluation interval, and clicking `Create`. | ||
|
|
||
| 9. Select the contact point (if none is defined, check its corresponding [section](#contact-point-creation)). | ||
|
|
||
| 10. (Optional) Configure the notification message. | ||
|
|
||
| 11. Click `Save`. | ||
|
|
||
| 12. Click `More` on the newly created alert rule, followed by `Export` and `With modifications`. | ||
|
|
||
| 13. Scroll down and click `Export`. | ||
|
|
||
| 14. Click `Download`, then move the file to the folder `grafana/provisioning/alerting/` inside the package for the SATP-Hermes project. | ||
|
|
||
| 15. Kill the running docker image for the container regarding otel-lgtm, rerun the test from step 2 and check if the alert is automatically provisioned. | ||
|
|
||
| 16. Go to the [`test file`](../src/test/typescript/integration/monitoring/functionality.test.ts) for monitoring and uncomment the commented lines that call the function `stopDockerComposeService`. |
65 changes: 65 additions & 0 deletions
65
...lugin-satp-hermes/grafana/provisioning/alerting/example-alert-#1-failed_transactions.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| apiVersion: 1 | ||
RafaelAPB marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # Definition of alert rule for failed transactions | ||
| # THIS IS AN EXAMPLE FILE. REPLACE THE VALUES WITH YOUR OWN ALERT CONDITIONS TO ENABLE ALERT NOTIFICATIONS. | ||
| groups: | ||
| - orgId: 1 | ||
| name: Alerts | ||
| folder: Alerts | ||
| interval: 1m | ||
| rules: | ||
| - uid: bf0keqynb8ni8d | ||
| title: Failed Transactions | ||
| condition: Failed Transactions over 0 | ||
| data: | ||
| - refId: Failed Transactions | ||
| relativeTimeRange: | ||
| from: 600 | ||
| to: 0 | ||
| datasourceUid: prometheus | ||
| model: | ||
| disableTextWrap: false | ||
| editorMode: builder | ||
| expr: failed_transactions_total | ||
| fullMetaSearch: false | ||
| includeNullMetadata: true | ||
| instant: true | ||
| intervalMs: 1000 | ||
| legendFormat: __auto | ||
| maxDataPoints: 43200 | ||
| range: false | ||
| refId: Failed Transactions | ||
| useBackend: false | ||
| - refId: Failed Transactions over 0 | ||
| datasourceUid: __expr__ | ||
| model: | ||
| conditions: | ||
| - evaluator: | ||
| params: | ||
| - 0 | ||
| type: gt | ||
| operator: | ||
| type: and | ||
| query: | ||
| params: | ||
| - Failed Transactions over 0 | ||
| reducer: | ||
| params: [] | ||
| type: last | ||
| type: query | ||
| datasource: | ||
| type: __expr__ | ||
| uid: __expr__ | ||
| expression: Failed Transactions | ||
| intervalMs: 1000 | ||
| maxDataPoints: 43200 | ||
| refId: Failed Transactions over 0 | ||
| type: threshold | ||
| noDataState: NoData | ||
| execErrState: Error | ||
| for: 1m | ||
| annotations: {} | ||
| labels: {} | ||
| isPaused: false | ||
| notification_settings: | ||
| receiver: test-contact-points | ||
22 changes: 22 additions & 0 deletions
22
packages/cactus-plugin-satp-hermes/grafana/provisioning/alerting/example-contact-points.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| apiVersion: 1 | ||
RafaelAPB marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # Definition of contact points for alerting | ||
| # THIS IS AN EXAMPLE FILE. REPLACE THE VALUES WITH YOUR OWN CONTACT INFORMATION TO ENABLE ALERT NOTIFICATIONS. | ||
| contactPoints: | ||
| - orgId: 1 | ||
| name: test-contact-points | ||
| receivers: | ||
| - uid: af1q2g33sy3uod | ||
| type: discord | ||
| settings: | ||
| # Replace with your Discord webhook URL (to test existing one, join the server using https://discord.gg/VnvMpcs2kT) | ||
| url: https://discord.com/api/webhooks/1430194325597196369/7Giic1fhAVEyhSO3Uy00yzgsyWMREEILBPMQACKXUghoB1KNdoOhoqWHntdE0CEanEJb | ||
| use_discord_username: false | ||
| disableResolveMessage: false | ||
| - uid: ff1q2g33sy3upc | ||
| type: email | ||
| settings: | ||
| # Replace with your email address | ||
| addresses: [email protected] | ||
| singleEmail: false | ||
| disableResolveMessage: false | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.