[monitor-config] feat: add generic Grafana dashboard composition framework

[a550580874]

What

A standalone, generic dashboard composition framework reviewed independently of any production migration. It contains the framework core, its packaged Jsonnet assets, the single optional CLI, tests, and evaluator setup; the total diff against main is 8 files, +1626/−0.

The runtime core lives in the installed package

• rl_insight/grafana/renderer.py is the single implementation of evaluation and serialization:
  • render_dashboards(config: Path) -> dict[str, Any]
  • generated_text(dashboard) -> str — deterministic serializer (sorted object fields from the composer plus fixed indentation)
  • materialize_dashboards(config: Path, output_dir: Path, *, overwrite: bool = True) -> list[Path]
  • stale_dashboards(config: Path, expected_dir: Path) -> list[Path] — the comparison behind --check
  • FRAMEWORK_DIR — the package path holding the generic Jsonnet assets
  • Failures raise JsonnetRenderError; its message always names the config path and keeps the Jsonnet stack trace.
• materialize_dashboards(..., overwrite=False) makes materialization purely additive: every target path is computed first, and if any of them already exists the call writes nothing at all and raises JsonnetRenderError naming the conflicting paths. The preflight runs before the first write, so a collision never leaves a partially materialized set behind. This is generic behaviour with no knowledge of any production naming; the default overwrite=True keeps the existing library and optional-CLI behaviour unchanged.
• rl_insight/grafana/__init__.py re-exports that API.
• rl_insight/config/services/grafana/jsonnet/framework/composer.libsonnet and viz.libsonnet — the generic Jsonnet assets now ship inside the package, so the wheel is self-contained and tools/ keeps no second source of truth. Both files are moved byte-for-byte (identical blobs to their previous tools/grafana/framework/ copies).
• composer.compose(modules, dashboard) provides explicit conflict rules for panel key/outputKey/id, owned rows, variables, and ordering references. ElementReference names resolve late from panel key to outputKey.
• Modules may define optional rowItems: { <existing row>: [GridLayoutItem, ...] }. These items append to a supported row in module order without creating or overriding it. Unknown or unsupported targets are eager errors; owned-row duplicates remain errors.
• viz.libsonnet contains engine-agnostic visualization defaults.

Documentation: introduced by the dependent production PR

• The canonical Grafana documents are tools/grafana/framework/README.md and tools/grafana/framework/README.zh-CN.md — the only Grafana documents the upstream tree keeps, one per language, each following the same structure (Overview, the four dashboard developer scenarios, and the framework extension points).
• They are introduced by the dependent production PR [monitor-config] refactor: split Grafana dashboards into reusable modules #173, which owns the composition set they document. Ownership moved only to keep this generic framework PR under the 2000-added-line review budget; the final upstream paths (tools/grafana/framework/README.md, tools/grafana/framework/README.zh-CN.md) are unchanged, and this PR's own diff carries no README. [monitor-config] refactor: split Grafana dashboards into reusable modules #173 now carries the maintainer-provided English and Chinese documents.
• They describe the configuration-first Jsonnet workflow, the four dashboard developer scenarios, and the coexistence of the bundled static dashboards with the Jsonnet-generated _jsonnet versions.
• They are the canonical home for the framework extension points (module interface, composition rules, rowItems, framework/viz.libsonnet defaults). No migration-only material appears in either document.

Evaluator: rjsonnet runtime dependency

• rjsonnet>=0.5.6 is a base runtime dependency in pyproject.toml. It replaces the gojsonnet Python binding plus the Windows jsonnet CLI fallback.
• rjsonnet publishes CPython ABI3 wheels for Windows (x86/x64), macOS (x86_64/arm64/universal2) and the common Linux glibc/musl architectures, so pip install rl-insight is enough on every platform: no Jsonnet CLI, no go install, no compiler.
• Evaluation is in-process. The renderer never spawns a subprocess and never calls a Jsonnet CLI.
• The Windows-only go install github.com/google/go-jsonnet/cmd/jsonnet@v0.22.0 step is removed from .github/workflows/monitor_unit_test.yml, and its path filter now also covers rl_insight/grafana/** and rl_insight/config/services/grafana/**.

CLI is a thin optional wrapper

• tools/grafana/framework/generate.py is the only CLI around the framework. It keeps --config, --out-dir, --check, --expected-dir and exit codes 0 / 1 / 2, but only parses arguments and calls the package core. It does not implement, copy or shell out to an evaluator, and it renders exactly the bytes the core produces.
• It is a convenience for shell use; library callers use rl_insight.grafana.renderer directly, and nothing in the runtime path calls it.

Semantics unchanged

• Composition order, duplicate/conflict checks, validated variableOrder/rowOrder, late ElementReference resolution, additive rowItems, viz defaults and deterministic rendering are all unchanged.
• No implicit overrides, no panel patches, no inheritance.
• Nothing under rl_insight/config/services/grafana/dashboards/ changes.

Testing

Validated at head ee2d2d6332e1ec01801ec2ddd32f4a6a5b217993 on macOS (arm64, CPython 3.12):

• Framework suite: 30 passed (pytest -q tests/monitor/ut/test_grafana_framework.py). The 25 tests from the previous round are preserved; the 5 new tests cover the default overwrite behaviour, additive materialization when no target exists, a single existing target aborting the run with JsonnetRenderError naming that path, preflight ordering (a collision on the second dashboard must not write the first one), and every collision being listed when several targets exist. None of them mentions any production dashboard name.
• Full monitor suite on this branch alone: 118 passed, 1 skipped (pytest -q tests/monitor/ut).
• On a disposable integration worktree of the two final heads ([monitor-config] feat: add generic Grafana dashboard composition framework #174 ee2d2d6 + [monitor-config] refactor: split Grafana dashboards into reusable modules #173 48f827c, merge never pushed): 132 passed, 1 skipped.
• Wheel build and isolated install: rl_insight-0.3.0-py3-none-any.whl contains rl_insight/grafana/renderer.py and rl_insight/config/services/grafana/jsonnet/framework/{composer,viz}.libsonnet; installed into a fresh venv (which pulled rjsonnet automatically), then imported and rendered a composition from a directory outside the repository, with jsonnet off PATH and gojsonnet/_gojsonnet absent. Import, asset presence and render all succeeded. The wheel contains no tools/, no tests/ and no migration script.
• pre-commit run --all-files (ruff, ruff-format, mypy, license, compileall): pass. git diff --check: clean.
• git diff main -- rl_insight/config/services/grafana/dashboards/: empty.
• Exact diff against main: 8 files, +1626/−0 — test_grafana_framework.py 646, viz.libsonnet 396, composer.libsonnet 274, renderer.py 157, generate.py 109, __init__.py 37, pyproject.toml 5, monitor_unit_test.yml 2. The two framework READMEs (375 + 384 lines) are no longer part of this PR; [monitor-config] refactor: split Grafana dashboards into reusable modules #173 introduces them with the maintainer-provided content (blobs e1557768… and 1709514e…), which is what brings this PR under the 2000-added-line budget without deleting any framework test or core code.

Windows is exercised by this PR's CI: the framework tests run there through the same in-process evaluator, with no CLI install step.

Relation to #173

#173 is the dependent production integration and now owns the canonical Grafana documentation (tools/grafana/framework/README.md + tools/grafana/framework/README.zh-CN.md; same final paths). It composes production content modules over this generic framework, stages the bundled static dashboards and materializes the Jsonnet dashboards beside them with overwrite=False. This PR contains no production module or composition; documentation ownership moved to #173 only to keep this PR's diff under 2000 added lines.

Merge order: #174 first, then #173 adds the README pair that documents this framework.

[tardis-key]

Thanks for your contribution.
If you expect this to define how the dashboard will be organized later and to guide developers in using this approach, it at least needs a README explaining: What is this? Why is it designed this way? And what should we do going forward?

[tardis-key]

1. Please provide a minimal framework/pipeline without modifying the existing files, sufficient to clearly explain the dashboard restructuring plan without depending on any large-scale changes.
2. Then bring the framework to next Wednesday’s community meeting and present it—unless you think the documentation alone is simple and clear enough.
3. Once that’s done, we can begin the large-scale file restructuring.

[tardis-key]

Back to the proposal itself: the current framework has really only organized the inference-related content of the two dashboards under verl, namely the vLLM and SGLang dashboards. If I wanted to reuse the trainer dashboards under verl, what would I need to do, and how much additional development work would it take?

In my view, this framework or pipeline should not focus on specific dashboards. Instead, it should abstract things into sub-dashboards A, B, C, and D, and then provide configuration options showing how A, B, C, and D can be combined to compose new dashboards.

[a550580874]

Thanks for the direction — done. The PR is reshaped into a standalone generic framework and marked ready for review again (head 877ab19, additive-only, nothing under rl_insight/config/services/grafana/ touched):

• Generic composer + generator: compose(modules, dashboard) with explicit composition and conflict rules (duplicates of panel key/outputKey/id, row/variable names, and unresolved ordering references are hard errors naming the entries); deterministic generate.py --config/--check via the gojsonnet Python binding — no Go CLI, no workflow edits.
• A/B/C/D minimal example: four toy modules; the same module A is composed as A+B and A+C+D with a different row order, and generic tests cover determinism, check mode, every conflict rule, cross-module reference resolution, and composition order.
• README (tools/grafana/framework/README.md) covers What/Why/How, the module interface, the composition rules, testing, and — on your reuse question — a "Going forward" section: reusing the verl trainer dashboards means extracting the trainer panels into one module file and writing a composition config; no framework changes are needed, and the extra work is limited to authoring modules, comparable to the per-engine split planned in [monitor-config] refactor: split Grafana dashboards into reusable modules #173.

Local verification: pytest -q tests/monitor/ut → 75 passed (no skips, no loosened assertions); generator --check byte-identical; pre-commit run --all-files clean. CI is running on the push.

Could you start with the README plus the A/B/C/D example to confirm the design reads right? On the community-meeting walkthrough — that's the member's call rather than mine to promise; if the docs and example turn out to be enough for you, we can skip it, and the member will arrange whatever format you prefer once you've had a look.

[a550580874]

Follow-up on the example question: we removed the committed toy example dashboards and their generated JSON to keep this PR lean (~1.2k added lines, 6 files) - the README now carries a short inline example with semantic modules, and the 13 tests build their inputs in tmp_path. The composition model and test coverage are unchanged, and no production dashboard was touched. All checks are green on the current head.