[DeepSeek-V4] index_topk=1024 sparse indexer can't recall verbatim lines from ~18k+ prompts (breaks agentic cwd/env recall)

[yhl-amd]

Summary

When DeepSeek-V4-Pro is served with the fp8 sparse indexer at the default
index_topk=1024, the model cannot reliably recall a verbatim line that is
present in its own prompt once the prompt reaches a realistic agentic size
(~18k+ tokens with many tool definitions). Instead of reading the line back it
hallucinates a plausible-but-wrong value.

This is isolated to recall (no tools involved): the exact answer is literally
in the system prompt, the model is asked to copy it verbatim, and it still fails.

Impact

Agentic clients (Claude-Code-style via /v1/messages, Codex via /v1/responses)
put the working directory / environment in the system prompt alongside ~20+ tool
definitions (~20–30k tokens total). The model then cannot recall its own cwd,
so it fabricates absolute paths (e.g. /sgl-workspace/..., /home/<random>/...),
runs shell commands against non-existent paths, loops, and either goes off-topic
or burns hundreds of K tokens without answering.

Environment

• Model: DeepSeek-V4-Pro, fp8 sparse indexer (main default, no fp4 flag)
• Serve: -tp 8 --kv_cache_dtype fp8 --hf-overrides '{"index_topk": 1024}' --enable_prefix_caching --enable_chunked_prefill --max-model-len 1048576 --gpu-memory-utilization 0.7 (gfx950 ×8)
• ATOM: main 0c7d50f9 + PR [Bugfix] DeepSeek-V4: content-addressed paged SWA fixes prefix-cache corruption (#1417) #1423 (paged SWA) + [Frontend] openai: multi-model tool-call parsing + reasoning (GLM / MiniMax-M3 / DSML) #1443 + [Bugfix] Cancel inference on client disconnect + fix non-stream request leak #1441 picked
• temperature=0, stream=false

Reproduction (self-contained)

Builds a ~18k-token system prompt = 700 distinct tool-like lines with one unique
needle buried ~55% in, then asks the model to copy it verbatim (no tools):

python3 - <<'PY'
import json,urllib.request
needle="/opt/acme-widgets/build-42/workspace"
V=["Creates","Deletes","Updates","Queries","Renders","Compiles","Streams","Validates","Encrypts","Schedules","Indexes","Archives","Migrates","Profiles","Balances"]
N=["orders","invoices","clusters","shards","payloads","sessions","tenants","widgets","pipelines","caches","registries","snapshots","manifests","tokens","routes"]
lines=[f"- tool_{i:04d}: {V[i%15]} {N[(i*7)%15]}. Params id:string,count:int,opts:object; returns handle." for i in range(700)]
mid=int(len(lines)*0.55)
env=f"\n# Environment\nYou have been invoked in the following environment:\n - Primary working directory: {needle}\n - Platform: linux\n\n"
system="# System\nYou are an assistant with the tools below.\n\n"+"\n".join(lines[:mid])+env+"\n".join(lines[mid:])
b={"model":"/shared/data/amd_int/models/DeepSeek-V4-Pro","max_tokens":400,"temperature":0,"stream":False,"system":system,
   "messages":[{"role":"user","content":"In the Environment section of your system prompt there is a line 'Primary working directory: <path>'. Reply with ONLY that absolute path, verbatim. Do not call any tool."}]}
req=urllib.request.Request("http://localhost:9700/v1/messages",data=json.dumps(b).encode(),
    headers={"content-type":"application/json","anthropic-version":"2023-06-01"})
d=json.load(urllib.request.urlopen(req,timeout=150))
c=d.get("content",[]);ans="".join(x.get("text","") for x in c if x.get("type")=="text").strip()
print("NEEDLE:",needle); print("ANSWER:",repr(ans[-120:]))
print("RESULT:","PASS" if needle in ans else "FAIL")
PY

Results

input tokens: 18383 (server log), temperature=0, 3 runs:

run 1  PASS  /opt/acme-widgets/build-42/workspace
run 2  FAIL  /opt/acme-widgets/                      (dropped tail segments)
run 3  FAIL  /opt/acme-widgets/warehouse-42/inventory  (hallucinated)

→ ~1/3 correct. Note it fails at temperature=0 and is non-deterministic
even there (tp8 + MoE + fp8 atomic-reduction noise), so re-run a few times.

Real-world reproduction with a captured Claude-Code /v1/messages request
(~21k tokens: system with the Primary working directory line + 26 real tool
defs), asking to copy that line verbatim, temperature=0: 5/5 wrong, e.g.
/shared/kfkq/DPO_OBP/, /dummy/for/claude/code, /shared/ccache.

Contrast: the same needle in a short (<~5k) or highly repetitive prompt
is recalled fine — the failure appears when the needle competes with a large
amount of diverse, information-rich content (tool defs) at index_topk=1024.

Analysis / question

The needle is one line in ~18–29k tokens; with CSA ~4:1 compression the committed
count exceeds index_topk=1024, so sparse selection appears to drop the needle's
compressed tokens and the model falls back to a memorized guess. This looks like
the sparse-indexer recall limit at the default index_topk.

• Is this expected for index_topk=1024, and should the default scale with
  context length (or be raised for agentic/long-context serving)?
• Does raising index_topk (e.g. 8192) restore verbatim recall at ~20k? (Have
  not confirmed on this build yet.)
• Independent secondary factor: at --enforce-eager-off + tp8, temperature=0
  is not bit-deterministic, which compounds the flakiness.

Happy to share the full captured 21k reproducer body and a topk A/B on request.

[yhl-amd]

Root cause found: the indexer query is Hadamard-rotated but the indexer K is not

TL;DR — this is not a "topk too low", CSA-compression, or kernel-precision limit. The DeepSeek-V4 lightning-indexer scores (Hq)·k instead of q·k: the index query goes through a Walsh-Hadamard rotation (rope_rotate_activation) but the index key written by the compressor does not. Since Hadamard H is orthogonal, training computes (Hq)·(Hk) = q·k; serving computes (Hq)·k, a scrambled dot product. This flattens the indexer scores so a distinct needle can't be ranked into the sparse top-k — invisible in dense (all rows attended), fatal in sparse.

Where (main 0c7d50f9)

• Query — atom/models/deepseek_v4.py Indexer.forward_batched calls rope_rotate_activation(...) (aiter aiter/ops/quant.py), doc: "Apply interleaved RoPE to trailing rope_dim, then Hadamard-rotate" (+ fp8 quant). → query is Hadamard-rotated.
• Key — Indexer.compressor = Compressor(..., rotate=True) → atom/model_ops/v4_kernels/fused_compress.py::fused_compress_attn. That kernel does RoPE + quant + cache-scatter but has no Hadamard (grep -i hadamard fused_compress.py → empty; the call site passes no rotate arg). self.rotate=True is stored on the Compressor and never applied. The Compressor docstring even flags it:

  !!!! QUANT NOT YET FUSED ... Indexer path (rotate=True): rotate_activation(kv); fp4_act_quant_inplace(kv, 32) ... Currently the fused kernel writes raw post-RoPE BF16 to kv_cache, skipping both.

How it was isolated (each step measured, not inferred)

Repro on DeepSeek-V4-Flash-FP8, tp4, --enforce-eager, fp8 kv (43L/index_topk=512; ~10.5k ctx already enters the sparse regime, n_committed≈2635 > 512). Needle = one verbatim line among 700 tool-like lines, temperature=0, plain Q&A. 10.5k → 0/3, 3.2k → 3/3.

Every decode-path component verified correct in isolation, which is what pointed the finger at the model-level indexer inputs:

• Decode attention kernel — ATOM_USE_TRITON_ATTN=0 (pure-torch reference attn) still 0/3 ⇒ not the attention kernel.
• csa_translate_pack (logical→physical) — dense routes all committed rows through the same translate and works ⇒ translate is correct for arbitrary indices.
• top_k_per_row_decode — standalone vs torch.topk on synthetic logits: exact match (overlap 512/512, 1024/1024) ⇒ selection kernel correct.
• deepgemm_fp8_paged_mqa_logits — vs its torch reference ref_fp8_paged_mqa_logits, passes at blocksize=32 (the serving value) and 64 ⇒ logits GEMM correct.
• indexer_k_quant_and_cache (write) + cp_gather_indexer_k_quant_cache (read) — aiter op_test k_err=0 s_err=0 scale_err=0 ⇒ cache write/read/scale correct.
• Cross-check — at decode, deepgemm (paged read) vs cp_gather + fp8_mqa_logits (gather read) on the same real cache: cos=1.00000, maxabsdiff=0.0, topk_overlap=512/512, identical argmax ⇒ the read path is bit-exact faithful; the indexer faithfully computes its scores, and the needle genuinely ranks outside the sparse top-k.

Then code inspection found the query/key Hadamard asymmetry above.

Fix confirmed

Correcting the mismatch (Hadamard-rotate K to match the query — rotate_activation on the gathered K before scoring) restores recall at the default index_topk=512:

context / needle pos	before	after (K Hadamard-corrected)
10.5k, pos 0.25 / 0.55 / 0.80 / 0.97	0/3	3/3 (all)
23.6k, pos 0.55	0/3	3/3

(As a side note this also explains the old index_topk sweep: topk=2048 of 2635 "recovers" simply because it keeps ~78% of rows — a band-aid, not a fix.)

Recommended fix (make q and k consistent; both give q·k since H is orthogonal)

• (A) preferred — drop the Hadamard on the index query (RoPE + fp8 only): one call site in Indexer.forward_batched, covers prefill and decode, deepgemm/fp8_mqa stay native. Cleanest as a no-Hadamard variant/flag of rope_rotate_activation.
• (B) — apply the Walsh-Hadamard to K in fused_compress_attn when rotate=True (matches the trained rotate=True semantics; later also port the fp4 QAT round-trip). Heavier (Triton kernel change).

Note _score_topk_prefill (uses fp8_mqa_logits on cp_gather'd un-rotated K + Hadamard'd q) has the same asymmetry, so (A) fixes both paths in one place.

Root cause + full measurements available; happy to open a fork PR for (A).