docs + grafana: paneles, alerting y pipeline pull-based - #92
Merged
Conversation
The publish pipeline now lives entirely in fcefyn_testbed_utils via the collect-lime-results.yml workflow that pulls report.xml artifacts from lime-packages instead of relying on a workflow change there. Replaces the old TESTBED_UTILS_TOKEN-based push diagram with the new pull sequence, lists artifact→path mappings, and documents required secrets (LIME_PACKAGES_TOKEN, BOT_PR_TOKEN) plus the repo-level toggles.
PAGES_BASE in dashboard.html points to this repo's Pages (fcefyn-testbed.github.io/fcefyn_testbed_utils/...), not lime-packages. Also remove the stale "publish-results job in build-firmware.yml" reference; reports now come from collect-lime-results.yml on this side.
Replace the TESTBED_UTILS_TOKEN rotation procedure with the two fine-grained PATs the new pipeline uses (LIME_PACKAGES_TOKEN for reading lime-packages artifacts, BOT_PR_TOKEN for opening the auto-merged PR). Expand the troubleshooting table with every failure mode we hit while wiring this up: org-level Actions toggle, repo-level auto-merge toggle, develop branch protection signed-commits gotcha, and the PAGES_BASE host mismatch that surfaced as 404 on Report links.
Place names use bananapi_bpi-r4 / openwrt_one (no _1 suffix), single Belkin entries are 24.10.6-only on the lab, and 25.12.2 currently runs only on QEMU. Add the physical mesh / mesh-pairs rows that the dashboard now surfaces.
Adds the lab-overview.json dashboard to the dashboards table and bumps the Orchestrator Host panel count to ~40 to reflect the new Lab Services and WireGuard sections.
Adds rows to the Orchestrator Host sections table describing the new "Lab Services" panels (labgrid-exporter, pdudaemon, ser2net via node_systemd_unit_state) and the "WireGuard (wg0)" tunnel panels (bandwidth, packets, errors, link status).
So the inventory of dashboard JSONs in the repo matches what Ansible actually provisions.
Per-section breakdown of lab-overview.json so the doc matches what Grafana actually shows: scrape up cards, uptime/CPU/RAM stat panels, consolidated device table, and the firmware/target table sourced from node_openwrt_info labels.
The duts-node.json dashboard has a per-interface node_network_*_errs_total / *_drop_total rate panel. Was missing from the section table.
The bpi_r4_1 / openwrt_one_1 place names never existed in the labgrid config — the actual places are bananapi_bpi-r4 and openwrt_one. Also drop the 25.12.2 physical rows (that release is QEMU-only on the lab right now) and add the physical mesh / mesh-pair entries that prepare_matrix.sh emits. The "Adding a new device" steps now point at the pull-based collect workflow.
\`bpi\` no longer matches anything — the place is bananapi_bpi-r4. Adds \`bananapi\` and \`openwrt\` as more useful examples that match the real labgrid place names.
Central to the collect-lime-results.yml pipeline, worth defining alongside the other CI/Labgrid terms.
Both metrics are referenced by the new Grafana panels (Lab Overview firmware table, Orchestrator Host Lab Services section) and worth calling out so the dashboard descriptions are self-contained.
Lists the two PATs the pipeline actually uses (LIME_PACKAGES_TOKEN, BOT_PR_TOKEN) so a reader skimming the glossary can find what secret does what without digging into publishing.md.
Mention that the same test-results-* artifacts surface on the CI dashboard via collect-lime-results.yml, so a reader debugging a failure knows where to find the post-hoc report view without digging through Actions artifacts.
Section 7 used to describe a publish-results job inside lime-packages that pushed report.xml into this repo via TESTBED_UTILS_TOKEN. That job never landed upstream. The replacement describes the inverse flow that ships from fcefyn_testbed_utils today: collect-lime-results.yml fetches artifacts, opens an auto-merged PR with BOT_PR_TOKEN, and pages.yml deploys. Also updates the sequence diagram.
…etail Replace the stale publish-results reference with the actual workflow that pulls report.xml into the dashboard and include the manual trigger instructions, so an operator hitting this case can act on it.
…hestrator Panel stat con 100 * node_nf_conntrack_entries / node_nf_conntrack_entries_limit. Relevante en el orchestrator porque mantiene autossh + SSH-proxy por DUT más los flujos de Prometheus/Grafana
Panel stat por DUT que lee node_load1, con thresholds en 1 y 2. Sirve para detectar DUTs trabajando bajo presión aunque el % CPU instantáneo se vea bajo, complementando el panel de CPU busy ya existente.
Panel stat por DUT que muestra el % de swap en uso: 100 * (SwapTotal - SwapFree) / max(SwapTotal, 1).
Sirve para ver de un vistazo si algún DUT está viendo tráfico inesperado durante tests o si quedó algo silencioso.
8 reglas para provisionar en Grafana (formato Grafana managed alerting v1): - fcefyn-target-down: target Prometheus inalcanzable > 2 min - fcefyn-high-cpu: CPU del orchestrator > 85% por 5 min - fcefyn-high-memory: RAM > 90% por 5 min - fcefyn-low-disk: filesystem raíz < 10% libre - fcefyn-tunnel-down: autossh DUT tunnel caído > 3 min - fcefyn-systemd-failed: alguna systemd unit en estado failed - fcefyn-network-errors: errores rx/tx > 0 sostenidos por 5 min se traen al flujo principal para que el Ansible role los provisione junto con los dashboards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.