docs: Address PR #381 review feedback

[rubambiza]

Summary

Follow-up to #381, addressing review feedback from Mike and Ansu on the benchmarking scenarios doc.

Actuation path and naming changes

• Rename lukewarm to "Cold Start (with launcher)": the path is cold, not warm. Metric renamed from T_luke_warm to T_cold_launcher.
• Remove M2 actuation path: focus the matrix entirely on M3 (launcher-based) paths.
• Rename T_launcher to T_instance_create: the metric measures instance creation, not launcher behavior (per Ansu's feedback).

Metric precision (end-instant clarity)

• T_actuation: explicitly ends at kubelet readiness probe on the requester pod (Pod Ready condition True).
• T_cold_launcher, T_instance_create, T_instance_ready: explicitly end at DPC readiness relay (V5 log: "Successfully relayed the readiness").
• Rename Hit_rate to Hot_hit_rate: parallels the new Warm_hit_rate metric.
• Add Warm_hit_rate: fraction of requests satisfied by an existing launcher pod.

Constituent metrics and observability

• Add constituent duration metrics table: T_launcher_schedule, T_launcher_startup, T_dpc_react, T_instance_ready as planned sub-metrics with observability sources.
• T_dpc_react: replace "negligible" with precise language; add note about later DPC actions (labels/annotations) being captured by T_instance_ready; mark observable as not yet present, link [Feature]: Add more logs on launcher/instance creation in dual-pods controller #495.
• T_instance_create start instant: link [Feature]: Log launcher receipt of request to create new instance #497 for subsecond launcher logging.
• Add alternative observability note: Prometheus histograms and distributed tracing (OpenTelemetry) as alternatives to log parsing.

Other fixes

• Add L2 to Resource Scaling and Stress Test: TTFT cost per requester is low relative to actuation time, and the data is useful at scale.
• Drop LPC attribution from Warm Start: DPC does not care whether the pre-existing launcher was created by LPC or by a prior DPC reconciliation.
• Fix node-level language: launchers are on Nodes, not GPUs.
• Change "launcher API responses" to "launcher logs" in constituent metrics preamble.
• Align Phase 2 descriptions with updated end-instant definitions.
• Remove unused L1+L3 legend entry; expand L1+L2+L3 description.

Test plan

• Verify markdown tables render correctly
• Confirm all 4 actuation path columns match between Paths table and Matrix
• Confirm metric names are consistent across definitions, L1 legend, and Integration Phases
• Verify issue links ([Feature]: Add more logs on launcher/instance creation in dual-pods controller #495, [Feature]: Log launcher receipt of request to create new instance #497) resolve correctly

Related issues

• [Feature]: Add more logs on launcher/instance creation in dual-pods controller #495 -- Add pre-CreateNamedInstance DPC log statement
• [Feature]: Log launcher receipt of request to create new instance #497 -- Add subsecond launcher logging for create request receipt

[MikeSpreitzer]

I left some individual comments.

[MikeSpreitzer]

I still think that it is important to be clear here about the fact that the launcher does not create the vllm instance on the launcher's own initiative but rather in response to a command from the DPC. This document still reads like there is just one DPC reaction time involved, and that is just not true. For example, one bit of path not discussed is from vLLM becoming ready to the server-requesting Pod being ready. That one goes like this: the DPC occasionally asks vLLM whether it is ready; after receiving a "yes", the DPC commands the requester container in the server-requesting Pod to start responding "yes" when queried for readiness; the kubelet occasionally queries that, and upon receiving a "yes" requests a kube apiserver to set that container's readiness; some time after that, something tells the kube apiserver to mark the whole server-requesting Pod as ready.

I would replace

then the launcher creates a new vLLM instance

with

then the DPC commands the launcher to create a new vLLM instance

[rubambiza]

Replacement is done! I appreciate the extra clarification, but I am trying to be mindful of this document's scope growing too much. In particular, I don't want to recreate the latency outline. Perhaps this doc can reference a companion doc that meets this need. I'd love to review a PR from you with doc-wide changes after this one is merged or modify this one myself once you have the latency outline completed. What do you think?

[MikeSpreitzer]

I will come back to this after we get this one merged.

[MikeSpreitzer]

If T_instance_create has only one constituent, why break it out?

[rubambiza]

We break it out because collapsing it into T_instance_ready would make the metric definitions ambiguous. T_instance_create is a primary metric that applies to both warm start and cold start (with launcher) paths. T_instance_ready is a constituent of the T_cold_launcher decomposition. In the warm start case they converge, but keeping T_instance_create as the top-level metric gives warm start a named metric without forcing it into the T_cold_launcher decomposition table.

[MikeSpreitzer]

| T_instance_ready | CreateNamedInstance call to DPC successfully relaying readiness to the requester pod (DPC V5 log: "Successfully relayed the readiness") | DPC logs + Kube pod status |

That does not say "warm start but not cold start" to me. Also, it is not precise about the starting instant.

[rubambiza]

You're right on both counts. T_instance_ready applies to both cold start (with launcher) and warm start. It's a constituent of T_cold_launcher in the former, and approximately equals T_instance_create in the latter. I'll clarify this in the definition. For the starting instant, "CreateNamedInstance call" means when the DPC issues the HTTP request to the launcher. This isn't precisely observable yet (tracked in #495); the closest current proxy is the launcher's HTTP access log timestamp for the PUT request.

[MikeSpreitzer]

I left some individual comments.

[MikeSpreitzer]

There are two variants of warm start: one just creates a vllm instance, the other first deletes a sleeping instance and then creates the new instance.
The implementation does not yet do the second, but it is coming (this is Issue #246).

[rubambiza]

Are you suggesting that we break them up into two distinct actuation paths or lump the cases but with an extended description (i.e., creating a new vLLM instance *or* delete an unsuitable sleeping instance and then create a new vLLM instance in an existing launcher pod?

[MikeSpreitzer]

I think that maybe we could go either way. If we call these both the same "actuation path" then, for purposes of understandable results, we would need a further level of distinction --- something like different cases within one actuation path. I suspect this might be the way to go, addressing #408 is going to also split the cold-start-with-FMA actuation path into two cases. Actually both could theoretically involve more than two cases (e.g., delete 0, delete 1, and delete 2) --- but I expect that deleting more than 1 will be very rare in practice, so am fine with not benchmarking them.

[rubambiza]

I think it makes sense to keep the lumped but insert an appropriate signal in the results metadata about which case applied (i.e., as simple as an extra label in the result string or something). I'd like to leave the implementation details for later.

[MikeSpreitzer]

OK with me.

[MikeSpreitzer]

Not a problem introduced in this PR, but: I see no reason to think that 100% hot-start hit rate is achievable. That would require always getting very lucky. And clearly, different usage patterns will have different hit rates.

[rubambiza]

Hence the close part. I think it would be rare, if not impossible, to hit 100% on active clusters like ours.

[MikeSpreitzer]

I do not think that "close to" does the job here. How good the hot-start hit rate is will depend on the overall usage pattern (over time) and how lucky is the Pod and GPU scheduling. These could easily be far from getting 100% lucky.

[MikeSpreitzer]

This is not the first interval to end at readiness, but it is the first one with a remark about how that is observed. At the earlier mention I assumed that the observation plan was the last-update timestamp on the readiness Condition in the Pod's status.

The observation plan should be stated at first mention of the moment, and/or in a list of observation plans.

[rubambiza]

I don't quite follow. What do you consider to be the remark about how it is observed? I stated it as (DPC V5 logs: ...), which is also shared with the earlier mention on T_cold_launcher. Perhaps you meant the observation for T_actuation? There, too, the readiness is quite descriptive (the kubelet's readiness probe on the requester pod succeeds, marking the Pod's Ready condition True), which aligns with your assumed observation plan about the last-update of the readiness condition.

[MikeSpreitzer]

Yes, the earliest mention of ending at requester readiness is on new line 39 with the text "the kubelet's readiness probe on the requester pod succeeds, marking the Pod's Ready condition True". That text has two different things in it, the second one is a process (a call from the kubelet to a kube apiserver) that covers a span of time, and both are different from what is said here ("DPC V5 log ...").

[MikeSpreitzer]

You want to be careful about each instant involved. For example, I see two different starting instants.

T_cold_launcher says it starts when "the DPC sends the request to create the launcher Pod".

T_launcher_schedule says it starts at "Launcher pod creationTimestamp".

[rubambiza]

Good catch. I've aligned both T_cold_launcher and T_launcher_schedule to start from the launcher Pod's creationTimestamp. The alternative would be a pre-creation DPC log statement (tracked in #495), but since Kubernetes condition timestamps (lastTransitionTime) are at second precision, using creationTimestamp (also second precision) keeps the decomposition internally consistent. The DPC-to-API-server gap is nanoseconds, so no meaningful accuracy is lost.

[MikeSpreitzer]

The DPC-to-API-server gap is nanoseconds

Do you have evidence of that? (Beware timestamps collected from different clocks. IIRC ntp typically gets you within about 1 ms.)

But regardless of whether it is milliseconds or nanoseconds, the point about information lost is that it is some unknown fraction of a second.

[rubambiza]

The Huygens clock synchronization work demonstrates the possibility to synchronize hosts to within tens of nanoseconds. However, I do not have such evidence for the DPC-to-API-server gap in particular.

[MikeSpreitzer]

I have finished a round of review.

[MikeSpreitzer]

Prometheus histograms are not strictly an alternative: they could be done additionally.

[MikeSpreitzer]

See comment on "alternative" above.

[MikeSpreitzer]

This still needs some work; I would not complain if it were done in follow-on PRs. Substantial progress has been made in this PR.