Consume centralized DPC "HTTP call done" log line; align actuation baseline - #1583
Conversation
Replace message-string dispatch with purpose-based dispatch on the new centralized "HTTP call done" klog line. The DPC now emits one HTTP-call line per request with purpose field identifying the call type (wake, create_instance, relay_ready). Ignores relay_unready and other purposes. Cold-launcher path unchanged (k8sCallStartTime). - Convert all test log lines to new centralized format - Implement purpose-based dispatch in parse_dpc_log - Update _DPC_INDICATOR_MSGS and module docstring - All 15 tests pass (hot/warm/cold paths + edge cases) Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Updated two robustness tests to use the new centralized "HTTP call done" format with purpose field instead of the old distinct messages. Preserves coverage for _is_dpc_log_file streaming behavior and partial-log handling. - test_indicator_message_past_256kb_still_parsed: Filler now uses benign purpose="get_health" HTTP call done lines (indicator but no timing record) to ensure file is recognized as DPC, while real anchor still lands past 256KB. - test_file_without_relay_but_with_wake_still_parsed: Updated to new format with full HTTP call done fields for wake-only case (requester crashed). All 15 tests pass. No production changes needed. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
The prior filler used "HTTP call done" purpose="get_health" lines, but
_is_dpc_log_file matches on the message string "HTTP call done" alone, so it
returned True at the very first filler line. That defeated the regression: a
re-introduced 256KB-sniffing bug would still find the indicator immediately.
Switch filler to a benign non-indicator klog line ("Flag" ...) so the first
indicator in the file is the real anchor from SAMPLE_LOG, sitting past the
256KB boundary. Same assertions; regression now genuinely bites.
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
The DPC's fma_actuation_seconds metric (llm-d#599) now measures actuation from the requester inference-server container state.running.startedAt, not the requester pod creation_timestamp. Align the harness Kube-timestamp fallback to the same baseline so it agrees with the controller. Replace FMALauncherInfo.dpc_timing_available (bool) with a per-iteration timing_source string: dpc | kube_container_start | kube_pod_create. dpc_timing_available is kept as a derived property (timing_source == 'dpc') and still emitted in dump() for backward-compatible downstream readers. Add get_container_start_timestamp() mirroring the controller read and select_kube_fallback_baseline(): prefer container start; when unavailable, revert to creation_timestamp, set timing_source=kube_pod_create, and emit a logger.warning naming the requester so a reversion is never silent. Propagate timing_source (and container_start_timestamp) through native_to_br0_1.py and render a three-way Source column in nop-analyze_results.py so kube_pod_create is distinguishable from kube_container_start. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
select_kube_fallback_baseline previously emitted the kube_pod_create 'may be overstated' warning inline whenever container_start_timestamp was 0.0. But the DPC-refine step in the finally block can later override timing_source to 'dpc', so a requester that ended up DPC-timed could still have emitted a spurious, mooted WARNING during the loop. Make select_kube_fallback_baseline pure (select baseline + tentative timing_source, no logging). Add warn_on_pod_create_baseline() and call it once, after DPC refinement, only for launchers whose FINAL timing_source is kube_pod_create. A genuine reversion still warns; a DPC-overridden one no longer does. kube_container_start / dpc paths are unchanged. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
| for launcher_info in iteration["launcher_infos"]: | ||
| ct = float(launcher_info["requester_info"]["creation_timestamp"]["value"]) | ||
| rt = float(launcher_info["requester_info"]["ready_timestamp"]["value"]) | ||
| ttrr = rt - ct if rt > 0.0 else 0.0 |
There was a problem hiding this comment.
Should this time also start at container_start instead of create?
Cluster e2e validation — passed ✅Ran standup → 10 iterations → teardown on FMA v0.6.0 with Qwen3-8B, harness built from this branch tip ( Emitted
Scope note (honest coverage boundary)Because every real actuation had DPC-log HTTP timing present, the run exercised only the Flipping to ready for review. |
Summary
Updates the FMA DPC-log timing pipeline for two coordinated upstream changes in
llm-d-fast-model-actuation(both present in its v0.6.0 release):Consume the centralized
"HTTP call done"log line. The dual-pods controller replaced its three per-message HTTP-timing log lines with one centralized line discriminated by apurposefield. The parser now keys onpurposetokens:wake→ hot-start anchor,create_instance→ warm-start anchor,relay_ready→ shared readiness end (relay_unreadyignored). Cold-launcher timing is unchanged (it comes from a Kubernetes API call, not an HTTP call). Without this, once the upstream change is picked up the parser silently falls back to coarse Kubernetes-timestamp upper bounds and the sub-second wake fidelity goes blank.Align the actuation baseline to container start. The controller's actuation metric now measures from the requester
inference-servercontainer'sstate.running.startedAtrather than the requester pod'screationTimestamp. The harness's Kube-timestamp fallback is realigned to the same baseline so it agrees with the controller.Closes #1547.
Key change: explicit, non-silent timing source
The single
dpc_timing_availableboolean is replaced with a per-iterationtiming_source. The three values measure three different intervals (not one interval at three fidelities) — each ends at requester readiness but subtracts from a different start point:dpc— measured inside the DPC log asrelay_readiness − httpCallStartTimeof the wake/create call. Baseline is the HTTP call start (after the container is up), so it is the tightest interval. Highest fidelity.kube_container_start— Kube fallback:requester ready − container state.running.startedAt. Matches the controller's own actuation-metric baseline; a coarser upper bound thandpc(starts earlier).kube_pod_create— Kube fallback reverted torequester ready − pod creationTimestampbecause container-start was unavailable. Earliest baseline, coarsest, degraded.A
kube_pod_createreversion emits alogger.warningnaming the requester (fired only for the final source, after DPC refinement, so a DPC-timed iteration never emits a spurious warning). Because the sources measure different intervals, per-iterationtiming_sourceexists so consumers don't mix them blindly. It propagates throughnative_to_br0_1.pyand renders as a three-way Source column in the analysis output;dpc_timing_availableis retained as a derived property for backward-compatible readers.Testing
tests/test_dpc_log_parser.py(15) +tests/test_fma_functions.py(18); full suite 562 passed, 31 skipped.t_hot(0.536 / 0.537 / 0.542 / 0.591 s, median 0.539s) — the same log yielded zero records under the old parser. Corroborates the ~0.51s vLLM-log weight-reload figure.Draft status
Marked draft pending cluster end-to-end validation: confirm the emitted
result.yamlshowstiming_source: dpcwith sub-secondt_wake, and that any fallback iteration reports the correct source. The smoke test above exercised the DPC-log path against real data; the container-start fallback path is unit-tested only, and is what the cluster run will exercise. Will flip to ready for review once that passes.