Install logging and monitoring tools on the Gateway.

[cesargithedgehog]

No description provided.

[cesargithedgehog]

Graphana Loki, and Prometheus stack. We need something to collect logs and metrics during development.

[cesargithedgehog]

We will use the LGTM stack for the gateway - Grafana and Prometheus. Our current model expects users to have their own LGTM stack or a compatible API solution. We will install Grafana Alloy (the agent responsible for data collection) on the switches and gateways.

At some point, we will support deploying the LGTM stack to some dedicated control node(s) since it can be very resource-intensive.

Grafana Alloy collects various metrics and logs and sends them to the LGTM APIs.
https://grafana.com/docs/alloy/latest/

We currently do not want to host it as part of the product because it is a large service to manage, and many people already use it.

*LGTM stands for Loki, Grafana, Tempo, and Mimir, which are the key components of this stack. The LGTM stack aims to provide a unified solution for monitoring and troubleshooting modern applications and infrastructure by addressing the three pillars of observability: logs, metrics, and traces

[daniel-noland]

It would be very helpful to have a mock stack set up in our development environment so that we can collect metrics and logs and start training our engineering staff on how to use them for debug and development.