fix: work around Triton buffer-store miscompile and ROCm compiled-training NaNs

[WangLingxun]

Why

Two classes of silent ROCm compiled-training failures showed up while validating this branch:

1. Triton buffer-store miscompile. The AMD Triton backend can emit a buffer_store_dwordx4 whose data VGPRs are redefined by a later instruction with no intervening s_waitcnt vmcnt, so the store writes whatever the clobbering instruction left in those registers. Nothing faults and nothing warns; a few percent of the kernel's output elements are simply garbage. Inductor's SwiGLU backward fusion trips this and poisons the MLP weight gradients, which diverges TorchTitan training. Megatron runs the same inductor-generated kernels, but there the corruption tends to stay inside the normal numeric range, so a finite loss curve does not prove a clean run.

2. TorchTitan compiled-training NaNs / invalid backward. Independently of Triton, ROCm Inductor graphs drop eager precision-cast truncation, DeepSeek MoE whole-block compilation poisons step-1 loss, and gfx942 aiter.fmha_v3_bwd incorrectly accepts DeepSeek MLA (qk=192, v=128) then launches an ASM kernel with incomplete metadata.

What changed

Triton per-kernel buffer-ops WAR (primus/core/patches/triton_bufops_war_patches.py). Wraps triton.compile, scans each kernel's emitted AMDGCN for the hazard, and recompiles only the affected kernels with buffer ops disabled. Keying on the machine code means there is no kernel or architecture allowlist to maintain. Registered for both torch backends at the build_args phase, gated only on being a ROCm GPU; non-HIP kernels are skipped per compile.

AMDGCN_USE_BUFFER_OPS=0 is deliberately not set globally: the hazardous kernels lose nothing without buffer addressing, but the Primus-Turbo grouped-GEMM kernels spill without it and carry no hazard, so the global switch costs ~18% on MoE recipes for kernels that never needed fixing. Per-kernel selection measured 299.9 vs 299.0 TFLOP/s per GPU on deepseek_v2_lite-BF16, against 244.6 with the global switch.

The patch also appends to TORCH_COMPILE_CACHE_KEY_TAG, since on an FX graph cache hit inductor never calls triton.compile — without invalidation a cache filled before this lands would keep serving hazardous binaries.

Divergence detection. tests/utils.py now requires every logged loss and grad norm to be finite, because a numerically diverged run still exits 0 and still prints the completion marker. tools/auto_benchmark/metrics.py marks a run with non-finite metrics invalid instead of publishing its throughput.

TorchTitan ROCm numerical workarounds.

• Enable Inductor emulate_precision_casts before compile so FSDP BF16 llama/qwen graphs keep eager truncation. This is a ROCm 2.12 BF16 issue, not a carry-over of upstream pytorch#150859.
• Restore TorchTitan v0.2.2's mixed compile strategy: dense TransformerBlocks stay fullgraph=True, DeepSeek MoE blocks stay eager. Whole-block MoE compile was putting EP dispatch/combine and FSDP hooks into one Inductor graph and producing step-1 NaN.
• Keep Turbo attention enabled on DeepSeek FP8. On gfx942, route MLA backward (qk=192/v=128) through aiter's generic CK path instead of the invalid fmha_v3 ASM dispatch. Forward is unchanged; measured ~204–206 TFLOP/s vs the original v26.5 CK-only path at ~198.
• Bind grouped-MM patches to the imported MoE module so PEP-420 namespace attributes no longer depend on prior test imports.

Tests. Hazard criterion and ROCm gate for the Triton WAR, finite-metric parser coverage for both backends' log formats, plus unit tests for precision-casts registration, dense-only MoE compile, gfx942 MLA CK routing, and Turbo converter gating. All unit tests are static: no GPU, no Triton.

Docs. A workarounds section in performance-tuning.md and one row in environment-variables.md.

Test plan

• Triton WAR + finite-metrics unit tests
• TorchTitan precision-casts / MoE compile / MLA CK / converter unit tests
• MI300X: hazardous kernel fixed and clean kernel untouched, residual 0
• MI355X: E2E green on both backends for the Triton WAR
• llama3.1 8B BF16 and qwen3 32B compiled training: finite loss/grad with precision-casts
• DeepSeek 16B BF16: step 1–3 finite after keeping MoE blocks eager
• DeepSeek 16B FP8: three finite steps via CK MLA backward, ~204 TFLOP/s, ~66 GiB
• CI E2E on both backends