[Single Controller / Async RL] cleanup tracking issue

[yuki-97]

Tracking issue for loose ends in the Single Controller / Async RL / Streaming.
Items will be checked off or split into separate issues as they land.

Known cleanup items

Drop after new implementation aligned with old implementation.

• Async RL
  • Drop nemo_rl/algorithms/async_utils/trajectory_collector.py .
  • Drop old implementation in nemo_rl/algorithms/async_utils/replay_buffer.py.
• Streaming
  • Drop nemo_rl/experience/rollouts.py, new implementation is in refactor: unify per-prompt rollout to RolloutManager #2567 and we'll no longer maintain sync rollout path.
  • Cleanup sync engine in inference backend, since we'll drop sync rollout path, so sync engine will no longer be used.

Known missing items

• Multi-reward support in AsyncRolloutImpl (dropped in feat: add AsyncRolloutManager for native async per-prompt rollouts #2566).

• greedy option in RolloutManager (dropped in refactor: unify per-prompt rollout to RolloutManager #2567).

• Fix env observation chat templating at the env interface level (flagged in feat: add AsyncRolloutManager for native async per-prompt rollouts #2566, context feat: Add Tau bench environment #2479).

• Epoch semantics in the Single Controller train loop: _train_pump was bounded only by max_train_steps, with no max_num_epochs/current_epoch notion of full dataset passes. — ✅ Fixed in feat(algorithms): SingleController streaming train_pump (split-API consumer) #2700 (feat(sc): epoch-bounded rollout passes (max_num_epochs parity), commit 4c2bf5b6c): SC now has max_num_epochs + a _current_epoch counter with epoch-bounded rollout dispatch, surfaced in run()/ping().

• Retry policy for the Single Controller train loop. _train_pump's per-cycle error path (single_controller.py L437-L573) aborts the worker step and re-raises on a mid-cycle failure (microbatch OOM, NaN-guard trip, prepare_logprobs / begin / finish failure), which currently kills the run — there is no retry. Flagged by the TODO(sc): retry policy is a follow-up at L441. (Flagged during feat(algorithms): SingleController streaming train_pump (split-API consumer) #2700 review.)

• Decouple logprob refresh from advantage config in the Single Controller. prepare_logprobs_from_meta is gated only on advantage_policy_logprobs_field/advantage_reference_logprobs_field, but prev_logprobs/reference_policy_logprobs are loss inputs (in DP_TRAIN_FIELDS), not just advantage inputs. A loss that needs them (PPO ratio, KL-in-loss) with a classical GRPO advantage cannot request the refresh without also feeding logprobs into compute_advantage. Add a loss-driven refresh flag independent of the advantage fields. (Flagged during feat(algorithms): SingleController streaming train_pump (split-API consumer) #2700 review, Known Missing Feature in feat(sc): train path — StalenessSampler + _train_pump rewrite #3220)

• Generation-level backpressure in the Single Controller — cap in-flight generations, not prompt groups. Today _rollout_pump backpressures per prompt group (max_inflight_prompts + the _buffer_capacity semaphore), so a group with one slow generation holds its buffer slot for the whole group → long-tail under-utilization (can't admit the next group even with spare generation capacity). Doing it properly needs the rollout/generation-worker rewrite that streams each generation into the DataPlane as it completes — today generate_and_push pushes a whole group atomically, so SC never sees individual generations finish and a generation-level semaphore has nothing to release against mid-group. Track with that rewrite; until then max_inflight_prompts stays as a conservative bound (under-utilizes on long tails but never over-subscribes DataPlane memory). (Discussed in feat(algorithms): SingleController streaming train_pump (split-API consumer) #2700, thread feat(algorithms): SingleController streaming train_pump (split-API consumer) #2700 (comment), Known Missing Feature in feat(sc): rollout path — TQReplayBuffer, rollout_manager, _rollout_pump #3219)

• Add over_sampling_ratio (>1) to avoid over sample to much and waste data. (Known Missing Feature in feat(sc): rollout path — TQReplayBuffer, rollout_manager, _rollout_pump #3219)

• Multi-mini-step inside a single RL step is not supported (one optimizer.step per RL step). (Known Missing Feature in feat(sc): train path — StalenessSampler + _train_pump rewrite #3220)

• Add custom sampler support — spilt TQ impl out and let people could easily rewrite their own custom sampler for select and evict logic. (Known Missing Feature in feat(sc): train path — StalenessSampler + _train_pump rewrite #3220)

• Drain gate in refit is not supported yet. (Known Missing Feature in feat(sc): setup + entrypoint #3266)

• invalidate_kv_cache only enabled in sync mode (max_staleness_versions=1), need to respect to recompute_kv_cache_after_weight_updates. (Known Missing Feature in feat(sc): setup + entrypoint #3266)

• Non-colocated inference with the Megatron generation backend is not supported in the Single Controller. setup.py asserts backend != "megatron" on the non-colocated cluster split, whereas legacy grpo.py supports it via a dedicated inference policy (init_megatron_generation non-colocated branch builds MegatronGeneration(cluster=inference_cluster, ...)). All SC recipes use vLLM generation so this is latent; either mirror grpo's non-colocated megatron-generation path or keep the fail-loud guard with a message that names the generation backend rather than reading like it's about Megatron training. (Flagged during feat(sc): setup + entrypoint #3266 review)

• FP8 KV-cache scale sync not implemented in the Single Controller. grpo's train loop, when the vLLM backend reports requires_kv_scale_sync (FP8 KV cache), computes FP8 QKV scales via policy.calibrate_qkv_fp8_scales(...) and forwards them through sync_weights(..., kv_scales=...) (feature). SC's _sync_weights calls sync_weights() with no kv_scales and never calls calibrate_qkv_fp8_scales, so an FP8-KV-cache rollout under SC keeps stale quantization scales after weight updates (degraded generation, cf. fix: forward calibrated KV cache scales on colocated ZMQ IPC refit path #3226). The synchronizer interfaces already accept kv_scales; only the driver-side compute+forward is missing. (Flagged during feat(sc): setup + entrypoint #3266 review)

• Router replay (R3) not supported/validated on the Single Controller (async + TransferQueue) path. The generation side is wired (configure_vllm_for_router_replay, gym require_routed_experts, TQPolicy field selection via fields_with_optional_routed_experts), but routed_experts are never preserved through the SC rollout → TQReplayBuffer → DataPlane → train path (that capture lives only in the legacy sync experience/rollouts.py and grpo's _preserve_router_replay_routed_experts), there is no SC-side fail-loud guard (unlike grpo.py's Async + TQ ... not been merged yet NotImplementedError), and there is no SC + R3 recipe/test. So policy.router_replay.enabled=true on an SC run silently trains without replayed routing. Split out into [Single Controller] Router replay (R3) not supported/validated on the async + TransferQueue path #3327. (Flagged during feat(sc): NeMo-Gym path #3267 review.)

• SC NeMo-Gym non-vLLM guard is unreachable dead code, placed after the expensive build. The if generation_config["backend"] != "vllm": raise NotImplementedError in setup_single_controller (setup.py:380) can't fire through the real entrypoint: sglang+gym trips _should_use_nemo_gym's should_expose_http_server assert and megatron+gym trips _build_generation's ValueError first; the guard only sits after _build_generation/_build_trainer (multi-minute cluster+model build). test_nemo_gym_rejects_non_vllm_backend reaches it only by mocking both earlier gates. Hoist the check to right after use_nemo_gym = _should_use_nemo_gym(...) (setup.py:327) so megatron+gym fails fast with a clear message before any build. (Flagged during feat(sc): NeMo-Gym path #3267 review.)

Add anything else here as it comes up.