This project is to describe how to transport data exported by LibreNMS into Prometheus through a Apche Kafka Broker.
To achieve this, we use the data pipeline software, Vector.
For now, this is a Proof Of Concept (no resiliency is provied). Especially, the completeness of all the metric types isn't guaranteed, as well as, that there isn't any bug in conversation. Maybe all possible value/conditions are yet to be handled.
Settings : I use 3 servers (VMs running Debian Linux 13) :
- One server for LibreNMS
- One server for Kafka and Vector
- One server for Prometheus (and Grafana not represented)
---
config:
theme: 'neutral'
---
flowchart TD
L(LibreNMS) e1@== Export Data with LibreNMS Kafka Exporter ==> K@{shape: das, label: "Kafka Topic"}
e1@{ animate: true }
K e2@== Getting Data ==> V1(Vector Kafka Sink)
e2@{ animate: true }
V1 e3@== Initiate Connection ==> K
e3@{ animate: true }
subgraph Vector
V1 e4@--> V2(Vector Transforms)
e4@{ animate: true }
V2 e5@--> V3(Vector Prometheus Exporter Sink)
e5@{ animate: true }
end
V3 e6@== Scraped Data ==> P1(Prometheus Scraper)
e6@{ animate : true }
P1 e7@== Initiate Scraping ==> V3
e7@{ animate : true }
subgraph Prometheus
P1 e8@--> P2(Prometheus Database)
e8@{ animate : true }
end
So LibreNMS is pushing data into a Kafka Topic.
Vector Kafka Source is pulling data from this topic and passes all these to a bunch of transform and routes components of Vector to handle all the type of messages/events/metrics LibreNMS produces. These transforms also handle the modification/correction so thaht metrics can be read by Prometheus Scraping (Upercase metrics name or name with hyphen - are forbidden).
All these metrics are concentrated into a single prometheus exporter sink.
Then Prometheus scraping process will then get these metrics and store them.
For getting state data from LibreNMS, we use standard LibreNMS API and scrape them Vector HTTP Client Source to expose these data to Prometheus. So that, both states and measurement are stored in Prometheus and can be dashboarded in Grafana.
State representation is simple : 0 = DOWN ; 1 = UP ...
When modding this part, beware of the cardinality of these when storing them in Prometheus.
---
config:
theme: 'neutral'
---
flowchart TD
subgraph LibreNMS
L(LibreNMS)
end
subgraph Kafka Broker
K@{shape: das, label: "Kafka Topic"}
end
L e1@== Export Data with LibreNMS Kafka Exporter ==> K
e1@{ animate: true }
K e2@== Getting Data ==> V1(Vector Kafka Source)
e2@{ animate: true }
V1 e3@== Initiate Connection ==> K
e3@{ animate: true }
V10(Vector HTTP Client Source ) e10@ == Initiate Connection ==> L
e10@{ animate: true }
L e10b@ == Geeting API-Pulled data ==> V10
e10b@{ animate: true }
subgraph Vector
V1 e4@--> V2(Vector Transforms for Kafka Data)
e4@{ animate: true }
V2 e5@--> V3(Vector Prometheus Exporter Sink for Kafka Data)
e5@{ animate: true }
V10 e11@--> V11(Vector Transforms for API Data)
e11@{ animate: true }
V11 e12@--> V13(Vector Prometheus Exporter Sink for API Data)
e12@{ animate: true }
end
V3 e6@== Scraped Data ==> P1(Prometheus Scraper for Kafka Data)
e6@{ animate : true }
P1 e7@== Initiate Scraping ==> V3
e7@{ animate : true }
V13 e13@== Scraped Data ==> P10(Prometheus Scraper for API Data)
e13@{ animate : true }
P10 e14@== Initiate Scraping ==> V13
e14@{ animate : true }
subgraph Prometheus
P1 e8@--> P2(Prometheus Database)
e8@{ animate : true }
P10 e15@--> P2
e15@{ animate : true }
end
It is also possible to split this configuration in two Vector Instances :
- One instance of vector pulled the API-based data from LibreNMS and sink them to a (dedicated ?) kafka topic
- One instance of vector get the data from the kafka topic(s) and expose them for Prometheus Scraping.
This is a basic configuration of LibreNMS to export Data into kafka topic
{
"enable": true,
"debug": false,
"security": {
"debug": ""
},
"broker": {
"list": "<YOUR BROKER HERE>"
},
"idempotence": false,
"topic": "<YOUR TOPIC HERE>",
"ssl": {
"enable": false,
"protocol": null,
"ca": {
"location": ""
},
"certificate": {
"location": ""
},
"key": {
"location": "",
"password": ""
},
"keystore": {
"location": "",
"password": ""
}
},
"flush": {
"timeout": 1
},
"buffer": {
"max": {
"message": 100
}
},
"batch": {
"max": {
"message": 10
}
},
"linger": {
"ms": 100
},
"request": {
"required": {
"acks": "0"
}
}
}Complete configuration of Vector :
[
LibreNMS2Kafka2Prometheus/vector-Kafka2Prometheus4LibreNMS.yaml
Lines 1 to 11 in 665d45d
Here is a representation of the Vector configuration :
%%{init: {'theme': 'base', 'themeVariables': {
'primaryColor': '#ffdfd3',
'edgeLabelBackground':'#fff',
'tertiaryColor': '#d3d3d3',
'nodeBorder': '#333',
'clusterBkg': '#f9f9f9',
'clusterBorder': '#bbb'
}}}%%
flowchart TD
%% ========== STYLES ==========
classDef source fill:#90EE90,stroke:#333,stroke-width:2px,color:#000;
classDef remap fill:#FFD700,stroke:#333,stroke-width:2px,color:#000;
classDef log_to_metric fill:#9370DB,stroke:#333,stroke-width:2px,color:#fff;
classDef route fill:#4169E1,stroke:#333,stroke-width:2px,color:#fff;
classDef lua fill:#FF8C00,stroke:#333,stroke-width:2px,color:#000;
classDef sink fill:#FF6347,stroke:#333,stroke-width:2px,color:#fff;
%% ========== SOURCES ==========
subgraph Sources["📡 **Sources**"]
A[("vector_metrics_source\n(type: internal_metrics)")]:::source
B[("kafka_metrics_source\n(type: kafka)")]:::source
C[("api_librenms_devices_source\n(type: http_client)")]:::source
D[("api_librenms_ports_source\n(type: http_client)")]:::source
end
%% ========== TRANSFORMS ==========
subgraph Transforms["🔄 **Transforms**"]
%% Remap
E["🟡 parse_metrics_tf\n(type: remap)"]:::remap
F["🟡 rewrite_measurement_type_tf\n(type: remap)"]:::remap
G["🟡 devices_split_tf\n(type: remap)"]:::remap
H["🟡 ports_split_tf\n(type: remap)"]:::remap
I["🟡 ports_add_status_tf\n(type: remap)"]:::remap
J["🟡 devices_convert_null_uptime_tf\n(type: remap)"]:::remap
K["🟡 metrics_downcase_poller_perf_tf\n(type: remap)"]:::remap
L["🟡 metrics_non_null_ports_tf\n(type: remap)"]:::remap
M["🟡 device_availability_remap_tf\n(type: remap)"]:::remap
%% log_to_metric
N["🟣 devices_l2m_status_tf\n(type: log_to_metric)"]:::log_to_metric
O["🟣 ports_l2m_status_tf\n(type: log_to_metric)"]:::log_to_metric
P["🟣 metrics_ports_tf\n(type: log_to_metric)"]:::log_to_metric
Q["🟣 metrics_poller_perf_tf\n(type: log_to_metric)"]:::log_to_metric
R["🟣 icmp_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
S["🟣 uptime_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
T["🟣 device_availability_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
U["🟣 mempool_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
V["🟣 storage_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
W["🟣 netstats_tcp_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
X["🟣 netstats_udp_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
Y["🟣 netstats_snmp_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
Z["🟣 netstats_ip_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AA["🟣 netstats_ip_forward_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AB["🟣 processors_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AC["🟣 hr_processes_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AD["🟣 hr_users_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AE["🟣 ipSystemStats_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AF["🟣 ucd_diskio_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AG["🟣 ucd_load_metrics_tf\n(type: log_to_metric)"]:::log_to_metric
AH["🟣 ucd_cpu_oid_tf\n(type: log_to_metric)"]:::log_to_metric
AI["🟣 ucd_cpu_non_oid_tf\n(type: log_to_metric)"]:::log_to_metric
%% Route
BA["🔵 route_metrics_tf\n(type: route)"]:::route
BB["🔵 ucd_cpu_route_tf\n(type: route)"]:::route
BC["🔵 reroute_non_rate_metrics_ports_tf\n(type: route)"]:::route
%% Lua
BD["🟠 change_gauge2counter_metrics_tf\n(type: lua)"]:::lua
end
%% ========== SINKS ==========
subgraph Sinks["📤 **Sinks**"]
CA[("metrics_prometheus_exporter_sink\n(type: prometheus_exporter)")]:::sink
CB[("devices_ports_api_prom_exporter_sink\n(type: prometheus_exporter)")]:::sink
CC[("vector_prometheus_exporter_sink\n(type: prometheus_exporter)")]:::sink
end
%% ========== LINKS ==========
%% Sources to Transforms
B -->|kafka_metrics| E
E -->|parse_json| F
F -->|rewrite_measurement| BA
C -->|devices| G
G -->|split_devices| J
J -->|convert_uptime| N
D -->|ports| H
H -->|split_ports| I
I -->|add_status| O
%% Route metrics
BA -->|poller_perf| K
BA -->|ports| L
BA -->|icmp_route| R
BA -->|uptime_route| S
BA -->|device_availability_route| T
BA -->|cpu_route| BB
BA -->|mempool_route| U
BA -->|storage_route| V
BA -->|netstats_tcp_route| W
BA -->|netstats_udp_route| X
BA -->|netstats_snmp_route| Y
BA -->|netstats_ip_route| Z
BA -->|netstats_ip_forward_route| AA
BA -->|processors_route| AB
BA -->|hr_processes_route| AC
BA -->|hr_users_route| AD
BA -->|ipSystemStats_route| AE
BA -->|ucd_diskio_route| AF
BA -->|ucd_load_route| AG
%% CPU route
BB -->|ucd_route_non_oid_route| AI
BB -->|ucd_route_oid_route| AH
%% Ports non-rate metrics
L -->|non_null_ports| P
P -->|enumerate_metrics| BC
BC -->|rate| CA
BC -->|_unmatched| BD
%% log_to_metric for devices/ports
N -->|device_status| CB
O -->|port_status| CB
%% Other log_to_metric
K -->|poller_perf| Q
Q -->|poller_metrics| CA
R -->|icmp_metrics| CA
S -->|uptime_metrics| CA
T -->|device_availability| M
M -->|remap_period| CA
U -->|mempool_metrics| CA
V -->|storage_metrics| CA
W -->|netstats_tcp_metrics| CA
X -->|netstats_udp_metrics| BD
Y -->|netstats_snmp_metrics| BD
Z -->|netstats_ip_metrics| BD
AA -->|netstats_ip_forward_metrics| CA
AB -->|processors_metrics| CA
AC -->|hr_processes_metrics| CA
AD -->|hr_users_metrics| CA
AE -->|ipSystemStats_metrics| BD
AF -->|ucd_diskio_metrics| BD
AG -->|ucd_load_metrics| CA
AH -->|ucd_cpu_oid_metrics| BD
AI -->|ucd_cpu_non_oid_metrics| CA
%% Lua transform
BD -->|change_gauge2counter| CA
%% Vector metrics
A -->|vector_metrics| CC
Sample Prometheus Configuration
Metric name of type counter doesn't comply with Prometheus practices. Metric of type counter should end with somethin like _total/ ... Some log_to_metric conversion have to improved for completeness