feat: Tighter FMA timing via DPC log parsing - #1504
Merged
Merged
Conversation
rubambiza
requested review from
Vezio,
achandrasekar,
kalantar,
maugustosilva,
mengmeiye and
namasl
as code owners
June 15, 2026 13:16
rubambiza
marked this pull request as draft
June 15, 2026 13:20
rubambiza
marked this pull request as ready for review
June 18, 2026 20:56
Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
The DPC timing refinement silently fell back to Kube upper bounds on real cluster runs. Two bugs, both surfaced during cluster validation: 1. _is_dpc_log_file() only sniffed the first 256KB of a .log for an indicator message. A real controller log was 1.08MB with its first relay/wake/create message at byte ~429K (after startup noise), so the log was discarded and the parser never ran. Now stream line-by-line with early-exit. Added a regression test with an indicator past 256KB. 2. native_to_br0_1 dropped the dpc_timing_available flag during conversion, so the analyzer's Source column always showed Kube even when timing was DPC-derived. Propagate the flag into launcher_infos. Validated on cluster: dpc_timing_available true on all iterations, hot-start t_wake ~0.53-0.59s (vs 8s Kube upper bound), Source=DPC. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
rubambiza
force-pushed
the
feat/dpc-log-parsing-timing
branch
from
June 19, 2026 16:33
1361393 to
5d06f1d
Compare
Collaborator
Author
|
The force push (5d06f1d) was a fresh rebase on main. |
aavarghese
enabled auto-merge (squash)
June 19, 2026 17:45
aavarghese
approved these changes
Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dpc_log_parser.pymodule that parses DPC controller logs to extracthttpCallStartTime/k8sCallStartTimefields at microsecond precisionbenchmark_fma()to override Kube-timestamp upperbounds with tighter DPC-derived values when available
verbosity, requester crash before relay), Kube upper bounds stand unchanged
dpc_timing_availablethroughnative_to_br0_1conversion so theanalysis report attributes each row to its true source
Closes #1465
Timing improvement (validated on cluster)
Measured on a 10-iteration
noprun (scale 0->1->0 per iteration). The DPC logparser now yields sub-second hot-start timing, confirming that the previous
8-10s figures were dominated by the kubelet readiness-probe interval rather than
actual wake latency.
t_wake)T_actuation)t_instance_create)An independent cross-check from the vLLM launcher log on the same run reported a
wake transition of ~0.51s, consistent with the DPC-derived
t_wake.Measurement conditions
meta-llama/Llama-3.1-8B-Instructghcr.io/llm-d-incubation/llm-d-fast-model-actuation/launcher:v0.6.0-alpha.13(includes PR Unexpected EOF from inference-perf-analyze_results.sh #522 timing fields)gpu_memory_utilization=0.95, prefix caching offNumbers are representative of a single validation run; hot-start hit rate (and
thus how often the sub-second path is exercised) depends on workload pattern and
launcher capacity configuration.
Test plan
Related
Assisted-By: Claude Code