feat: FMA actuation path classification, hit rates, and per-path timing by rubambiza · Pull Request #1429 · llm-d/llm-d-benchmark

rubambiza · 2026-05-29T20:01:28Z

Summary

Replace pod-name heuristic for actuation path classification with timestamp-based logic (launcher creationTimestamp vs requester creationTimestamp)
Rename T_LUKE_WARM to T_COLD_LAUNCHER to align with updated FMA terminology
Add Hot_hit_rate, Warm_hit_rate, cold_launcher_rate per iteration
Add upper-bound per-path timing: T_wake (hot), T_instance_create (warm), T_cold_launcher (cold)

Test plan

Run FMA scenario with hot-start config and verify T_hot classification + t_wake populated
Run FMA scenario with warm-start config and verify T_warm classification + t_instance_create populated
Run FMA scenario with cold-start config (no pre-existing launcher) and verify T_cold_launcher classification + t_cold_launcher populated
Verify hit rates sum to 1.0 per iteration
Verify JSON output includes new fields without breaking existing consumers

…timing Phase 1: Replace pod-name heuristic with timestamp-based classification. Launcher creationTimestamp vs requester creationTimestamp determines warm (pre-existing launcher) vs cold-with-launcher (DPC created new). Add Hot_hit_rate, Warm_hit_rate, cold_launcher_rate per iteration. Phase 2: Compute upper-bound per-path timing using Kube timestamps: T_wake (hot), T_instance_create (warm), T_cold_launcher (cold). Rename T_LUKE_WARM to T_COLD_LAUNCHER across harness and analysis. Closes llm-d#1422 Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

- Rename cold_launcher_rate to cold_launcher_hit_rate - Add per-path timing (T_hot, T_warm, T_cold) to analysis summary table - Add launcher node ID to results and analysis output - Display all hit rates (hot, warm, cold_launcher) instead of just hot - Add sleeper_limit to scenario metadata and analysis output - Handle None values in per-path timing columns (show "--") - Extend benchmark_report conversion for new FMA fields - Fix LLMDBENCH_FMA_SLEEPER_LIMIT not being read from env Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

aavarghese

One more thing: can we change https://github.com/llm-d/llm-d-benchmark/blob/main/config/templates/jinja/24_fma-deployment.yaml.j2#L47 to the new field name maxInstances: 4?

Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

- Change LauncherConfig template from maxSleepingInstances to maxInstances - Add maxInstances default (4) in defaults.yaml under fma.launcher - Add LLMDBENCH_FMA_MAX_INSTANCES env var to harness pod - Display Max Instances in analysis output (replaces Sleeper Limit display) - Propagate max_instances through scenario metadata and benchmark_report Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

rubambiza · 2026-06-09T20:22:20Z

Addressed in commit 23beecc:

Changed maxSleepingInstances: 3 → maxInstances: {{ fma.launcher.maxInstances }} in the LauncherConfig template (24_fma-deployment.yaml.j2)
Added maxInstances: 4 default in defaults.yaml under fma.launcher
Added LLMDBENCH_FMA_MAX_INSTANCES env var to harness pod template
max_instances now appears in scenario metadata, benchmark_report, and analysis output (replaces Sleeper Limit display with Max Instances)

Validated on cluster -- max_instances: 4 correctly reported in both raw and benchmark_report results.

sleeperLimit is an M2-only DPC config that doesn't affect M3 (launcher-based) actuation paths. Remove it from the harness env vars, scenario metadata, and benchmark report conversion. The Helm chart still passes it to the DPC for M2 compatibility. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>

rubambiza marked this pull request as ready for review June 4, 2026 14:04

rubambiza requested review from Vezio, achandrasekar, kalantar, maugustosilva, mengmeiye and namasl as code owners June 4, 2026 14:04