TITO prefill storms cause GPU contention that starves concurrent decode requests

[DavidBellamy]

Context

Analysis of Harbor RL training jobs revealed httpx.ReadTimeout errors in the session server proxy. Initial hypothesis was thundering herd at rollout batch boundaries, but further investigation showed a different root cause.

Analysis

We reproduced the timeout with a minimal job: 2 tblite tasks, 2 rollouts, single node, 8 DP shards. No thundering herd possible with 4 concurrent requests. The timeouts still occurred.

We believe the root cause is GPU contention between prefill and decode within a shared engine, not request flooding:

1. When a TITO session accumulates a long conversation (50K-100K+ tokens) and starts a new turn on a different DP shard, the full context must be prefilled from scratch
2. This prefill monopolizes the GPU for 30-40 seconds (visible in logs as 80+ consecutive 512-token prefill batches on one DP shard)
3. During the prefill storm, all other DP shards' decode throughput drops from ~68 tok/s to 3-8 tok/s
4. Already-long-running decode requests accumulate enough stall time to exceed the proxy timeout

The burst pattern observed in larger jobs (300+ errors at batch boundaries) is this same mechanism amplified: more concurrent requests means more TITO sessions starting simultaneously, more cross-shard prefills, more GPU contention.

Evidence from our job logs

Single engine, DP5 decoding at 68 tok/s. DP7 starts a massive TITO prefill:

• 19:05:19: DP7 begins 512-token prefill batches (80+ consecutive)
• 19:05:24: DP5 throughput drops to gen throughput (token/s): 4.41
• 19:05:33: DP5 at gen throughput (token/s): 3.64
• 19:05:44: DP5 at gen throughput (token/s): 3.45
• Prefill continues for ~35 seconds

This directly causes the in-flight decode request to stall, pushing it past the 600s timeout.

Recommendation

1. Primary fix: Restructure the timeout to measure engine processing time, not wall-clock time from submission (filed separately as issue Session server timeout should measure engine processing time, not wall-clock time from submission #936)
2. Secondary fix: Investigate whether chunked prefill can interleave with decode batches to prevent the GPU monopolization during TITO session prefills
3. Batch staggering is useful as a mitigation at scale (reduces the number of simultaneous TITO prefills) but is not the root cause fix

/cc @mingshanhee

[mingshanhee]

After examining and understanding the codebase more, I realised the issue might be more related to routing affinity (i.e., using the same inference worker for each request)

In TITO-based multi-turn agentic workflows, performance depends not only on token reuse but also on routing stability. If successive turns of the same session are sent to different SGLang inference engines, the new engine cannot reuse the previous cache state and must prefill the entire accumulated context again. This leads to the large unnecessary prefill time, which significantly hurt serving stability for long conversations.

• Issue 1: The session-server path does not attach a stable routing key when proxying /v1/chat/completions, unlike the /generate path, which can use X-SMG-Routing-Key for consistent routing.
• Issue 2: Miles Router is not sticky. It routes each request to the worker with the fewest active requests, rather than pinning a session to the previously used SGLang engine.

As a result, a long TITO session may hop to a different DP shard on a later turn. When that happens, the new shard lacks the prior cache state and must prefill the full context from scratch. For very long conversations, this creates avoidable prefill spikes, slows decode throughput, and increases timeout risk.

• Resolution: Add a session-level routing ID, similar to X-SMG-Routing-Key in SGLang Router, to session-server /v1/chat/completions requests. This preserves routing affinity so later turns return to the same SGLang inference engine whenever possible, reducing unnecessary full-context prefills. This should reduce the prefill time, and avoid hassle of "timeout to measure engine processing time, not wall-clock time from submission".
• Note: This will probably need applying some patch to sglang, similar to the patch enabling TITO with OpenAI-compatible endpoints.

[DavidBellamy]

Verbose logging data from a 33-node test run (job 1529054)

Ran a 1h25m async RL training job with verbose logging enabled (MILES router verbose, logging.DEBUG, NCCL_DEBUG=info) to capture the full timeline of a timeout storm. Results strongly confirm the TITO prefill contention hypothesis.

Key numbers

Metric	Count
httpx.ReadTimeout	625
LiteLLM 901s timeouts (300s x 3 retries)	230
Prefill throughput < 1 tok/s	152
Successful completions	3,793
Total errors	1,543
Training steps completed	1 (stuck on step 1 for entire run)

Prefill throughput during storms

During timeout storms, prefill throughput on affected engines drops to near-zero:

input throughput (token/s): 0.06   # effectively frozen
input throughput (token/s): 0.08
input throughput (token/s): 8.28   # severely degraded (normal: 100K-600K)

Meanwhile, engines handling fresh sessions (0 cached tokens) prefill normally:

input throughput (token/s): 624385.16   # fresh session, no contention
input throughput (token/s): 461635.39

The bimodal pattern is consistent: long-running sessions with 40-48K cached tokens drop to 5-15 tok/s on prefill, while fresh sessions are fine. This confirms the contention is from large-context TITO prefills starving concurrent requests on the same GPU.

Storm pattern

Timeouts occur in bursts of 30-100, then partially self-resolve as the blocking prefills complete:

• Storm 1: 18:43-18:46 (~100 timeouts)
• Storm 2: 18:52-18:54 (~30 timeouts)
• Chronic trickle: ~3-8/min between storms

Weight update correlation

Only 1 weight update occurred during the entire run (at 18:32, took 15.5s). The first timeout storm started at 18:43, 11 minutes later. Weight updates were not the trigger for this run's timeouts -- this is purely prefill contention.

Cascade effect

The timeouts trigger a cascade of secondary failures:

1. httpx.ReadTimeout (625x) -> 500 Internal Server Error
2. Session state corruption -> rollback failed: no assistant message found (127x)
3. Session eviction -> session not found 404 (99x)
4. Client disconnects while waiting (312x)
5. Trials fail with AgentError, reward=0.0

The 29% error rate (1,543 errors vs 3,793 successes) prevented the training loop from accumulating enough rollouts to progress past step 1.