Speculative prefill in front of dflash. C++/CUDA only.
A drafter loaded in-process scores token importance; the heavy target only prefills the spans that matter.
Qwen3.6-27B Q4_K_M at 128K on a single RTX 3090: 24.8 s TTFT vs ~257 s llama.cpp = ~10.4×, NIAH retrieval preserved.
Blog post · Discord · lucebox.com
Cold TTFT (s) Speedup NIAH
llama.cpp pp131072 ~257 1.0x ✓
dflash daemon @ 128K 24.8 10.4x ✓
dflash daemon @ 64K 13.5 10.0x ✓
Long context turns prefill into the dominant latency on quantized 27B targets. Speculative prefill scores token importance with a small drafter, then the heavy target only prefills the spans that matter. Quality preserved on NIAH at every measured context. The whole thing runs as a single C++/CUDA binary: no Python, no Triton, no PyTorch at runtime.
Long-context prefill is O(S²): vanilla llama.cpp on a single RTX 3090 takes ~257 s to prefill 131,072 tokens of Qwen3.6-27B Q4_K_M (FA on, Q4_0 KV). Decode after that is fast (dflash spec decode runs at ~74 tok/s) but the user is staring at a blank screen for 4 minutes before the first token.
Cross-Family Speculative Prefill (SambaNova ICLR 2026, Liu et al.) showed a small drafter can score per-token importance over a long prompt and select a tiny fraction without losing the needle. The reference impl (Jingyu6/speculative_prefill) wires this on top of vLLM with full BF16 targets on big GPUs.
What was missing: no implementation that sits in front of a quantized GGUF target on a 24 GB card without dragging Python+Triton into the runtime path. PFlash is that:
- C++/CUDA daemon-resident drafter + scoring + target generation, all in one process, one ggml allocator.
- Custom Qwen3.5-0.8B BF16 forward (
src/pflash/qwen35_loader.cpp+ the qwen35 target graph) — no libllama. - Scoring runs the drafter's blocks 0–14 (Gated DeltaNet layers plus a few full-attention layers on ggml flash-attention), then block 15's NoPE Q/K attention mass from the query window, optionally through a trained scoring head. No sparse attention kernel is involved.
- 128K → 2.6K span selection at
keep_ratio=0.05, NIAH retrieved at every measured context, decode ~74 tok/s downstream.
PFlash v1 (the Qwen3-0.6B drafter with a BSA sparse forward, replaced by the Qwen3.5-0.8B scorer in #746): NIAH single-needle, RTX 3090 24 GB, Qwen3.6-27B Q4_K_M target, keep_ratio=0.05.
| Source S | dflash TTFT | llama.cpp baseline | Speedup | NIAH |
|---|---|---|---|---|
| 64K | 13.5 s | 134.95 s (FA off, dense) | 10.0× | ✅ |
| 128K | 24.8 s | ~257 s (FA on, Q4_0 KV) | ~10.4× | ✅ |
Decode after prefill: ~74 tok/s (dflash spec decode + DDTree). The pipeline is the luce binary on its own — no Python in the inference loop.
PFlash is the algorithm. The implementation lives in ../../server/ as part of the luce daemon. The optimizations/pflash/ directory in this repo only contains the Python tooling for benchmarking (NIAH case generation, bench harness around the daemon stdin protocol). Production deploys hit the luce daemon directly.
# 1. from the repo root, install Python deps and build the server
cd lucebox-hub
uv sync
git submodule update --init --recursive
cmake -B server/build -S server -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CUDA_ARCHITECTURES=86
cmake --build server/build --target test_dflash -j
# 2. fetch weights (target + spec-decode draft + drafter scorer)
uv run hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir server/models/
uv run hf download Qwen/Qwen3.5-0.8B model.safetensors tokenizer.json --local-dir server/models/drafter/
uv run hf download z-lab/Qwen3.6-27B-DFlash model.safetensors --local-dir server/models/draft/
# 2b. convert the drafter (Qwen3.5-0.8B HF) to a BF16 GGUF for the C++ scorer.
# The submodule already vendors llama.cpp at deps/llama.cpp.
uv run python server/deps/llama.cpp/convert_hf_to_gguf.py server/models/drafter \
--outtype bf16 --outfile server/models/Qwen3.5-0.8B-BF16.gguf
# 3. generate NIAH cases + run head-to-head bench against the C++ daemon
uv run --directory pflash python tests/niah_gen.py --n 1 --ctx 131072 --out /tmp/niah_128k.jsonl
uv run --directory pflash python tests/bench_niah_cpp.py \
--bin ../server/build/test_dflash \
--target ../server/models/Qwen3.6-27B-Q4_K_M.gguf \
--draft-spec ../server/models/draft/model.safetensors \
--drafter-gguf ../server/models/Qwen3.5-0.8B-BF16.gguf \
--cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256For an OpenAI-compatible server with transparent compression on long prompts, run luce_server with these flags:
| Flag | Choices / type | Default | Effect |
|---|---|---|---|
--prefill-compression |
off / auto / always |
off |
When to run pflash. auto compresses when total prompt ≥ threshold; always compresses every request. |
--prefill-threshold |
int (tokens) | 32000 |
Token threshold for auto mode. 16384 with a trained scoring head. |
--prefill-keep-ratio |
float (0, 1] |
0.05 |
Fraction of source tokens to keep after compression. 0.02 for 128K, 0.10 for 32K. With a trained scoring head and no ratio flag, the release curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225 applies. |
--prefill-drafter |
path to .gguf |
required when not off |
Drafter weights (Qwen3.5-0.8B BF16 GGUF). |
--prefill-drafter-tokenizer |
HF repo id | Qwen/Qwen3.5-0.8B |
HF tokenizer for the drafter vocab. |
With PFLASH_SCORING_HEAD_GGUF set, every unset PFlash setting takes the measured release profile; see PFlash configuration.
./server/build/luce_server server/models/Qwen3.6-27B-Q4_K_M.gguf \
--draft server/models/draft/model.safetensors \
--max-ctx 8192 --fa-window 0 \
--prefill-compression auto \
--prefill-threshold 4096 \
--prefill-keep-ratio 0.02 \
--prefill-drafter server/models/Qwen3.5-0.8B-BF16.ggufBelow the threshold the server runs the standard target generate (no compression). Above it, the server transparently runs compress on the daemon, swaps the prompt for the compressed text, and continues the normal /v1/chat/completions flow. Tool-calling requests (req.tools non-empty) skip compression so JSON tool definitions stay intact.
Validated end-to-end at 64K and 128K source on RTX 3090 (Qwen3.6-27B Q4_K_M target + Qwen3.5-DFlash draft + Qwen3-0.6B BF16 drafter).
The dflash daemon runs persistently and accepts these commands on stdin (newline-delimited):
| Command | Effect |
|---|---|
compress <ids.bin> <keep_x1000> <drafter.gguf> |
Drafter scores the prompt and emits the compressed token-id stream (terminated by -1). |
generate <prompt_ids.bin> <n_gen> <out_ids.bin> |
Target spec-decode on the (already compressed) prompt. Streams committed token ids on stdout. |
park draft / park target / park |
Free draft / target / both weights from VRAM. |
unpark draft / unpark target / unpark |
Restore weights from disk to VRAM. |
free drafter |
Release the spec-prefill drafter context (drafter weights + KV). |
Typical flow at 128K on a 24 GB card: park target → compress → free drafter → unpark target → unpark draft → generate → park draft.
pflash.dflash_client.DflashClient is the Python wrapper around this protocol used by tests/bench_niah_cpp.py.
PFlash's selection, query, chat-view and evaluation switches are listed in PFlash configuration. On the daemon process:
| Env var | Default | Purpose |
|---|---|---|
LUCE_FA_WINDOW |
0 (set by the bench client) |
Sliding window of the target's full-attention layers; 0 keeps full attention on the compressed prompt. Unset, test_dflash uses 2048. |
LUCE_KV_K / LUCE_KV_V |
(auto) | KV-cache quant types. q4_0 / q4_0 is the bench setting. tq3_0 saves another ~4 GB at 128K. |
The LUCE_FP_* FlashPrefill switches do not affect PFlash: the scorer does not run the FlashPrefill kernels.
prompt (≤ 128K tokens)
│
▼
┌──────────────────────────────────────────────┐
│ drafter (in-process) │
│ custom Qwen3.5-0.8B BF16 forward in ggml │
│ blocks 0-14 (GDN + full attention) │
│ block 15 Q/K attention mass → score [S] │
│ chunks + keep-ratio budget → kept spans │
└──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ compressor (in-process) │
│ keep top keep_ratio of source tokens │
│ re-emit compressed token-id stream │
└──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ dflash spec decode (in-process) │
│ target prefill of compressed prompt │
│ DDTree spec decode + rollback │
│ → answer tokens │
└──────────────────────────────────────────────┘
Drafter forward. The Qwen3.5-0.8B drafter (src/pflash/qwen35_drafter.cpp on the qwen35 target graph) runs the model's first fifteen blocks and scores the context with block 15's NoPE Q/K attention-mass head; PFLASH_QWEN35_LEGACY_SCORER=1 selects the all-layer running-max scorer instead.
Scoring + selection. Each token's score is the attention mass the query window puts on it at block 15 (or the trained head's score). Chunks are taken by relevance until the keep-ratio budget is spent, after the structurally required spans (instructions, the query, the generation prompt). The budget comes from --prefill-keep-ratio or the --prefill-curve keep-ratio curve.
Memory budget on 24 GB. Drafter scoring at 128K needs ~7-10 GB (drafter + KV + scratch). Target + draft idle is ~18 GB. They can't coexist. The daemon's park / unpark / free drafter commands sequence VRAM occupancy across the request:
1. park draft + target # daemon idles at ~3 GB
2. drafter loaded + scored # ~10 GB peak
3. free drafter # release drafter weights + KV
4. unpark target # ~16 GB
5. unpark draft # +draft weights for spec decode
6. generate # spec decode the compressed prompt
7. park draft (idle) # back to target only
The algorithms are not ours:
- Cross-Family Speculative Prefill (SambaNova ICLR 2026): max-mean attention aggregation over a small drafter, lookahead-only attention.
- Speculative Prefill (Liu et al, 2025): the original Q-hook construction. Reference impl: Jingyu6/speculative_prefill.
- FlashPrefill (Fan et al, 2026): block-sparse attention with sink + window + dynamic top-K blocks. Original kernel: qhfan/FlashPrefill (Triton).
What we built:
- C++/CUDA port of the FlashPrefill algorithm (4 kernels,
mean_K / score / select / sparse_fwd, no Triton) and BSA (mit-han-lab/Block-Sparse-Attention) wired withoutlibtorch. They ran the v1 drafter forward; today Gemma 4's sliding-window prefill uses them, and the PFlash scorer does not. - Custom Qwen3.5-0.8B BF16 forward so the drafter runs through the same ggml allocator as the 27B target.
- Daemon stdin protocol (
compress/generate/park/unpark/free drafter) so target + drafter coexist on a 24 GB card. - NIAH harness against
llama-benchfor end-to-end validation.
- Single 24 GB GPU target (RTX 3090 reference). On 32+ GB cards, drafter + target can coexist and the park/unpark dance disappears.
- Qwen3.6-27B Q4_K_M target + Qwen3.5-0.8B drafter is the validated pair. Other targets/drafters need keep-ratio re-calibration.
- NIAH single-needle is the only retrieval task validated end-to-end. Multi-doc QA, long-form code retrieval, etc. still TBD.
These are small operational lessons collected while running PFlash as the long-context lane of an OpenAI-compatible service in front of Lucebox. Nothing here changes the published kernels or the in-process daemon protocol — they are tuning hints for production operators.
PFlash compress on a 64K prompt takes ~24 s end-to-end on a live
Qwen3.6 lane (RTX 6000 Ada sm_89, Q4_K_M target, Qwen3-0.6B
drafter). The default queue budget
(max_queue_requests=4, queue_timeout_s=12.0) was tuned for
shorter prompts and produced avoidable timeouts on bursts of long
prompts.
For PFlash long-context lanes, recommend tuning:
max_queue_requests=8queue_timeout_s=90.0
These are operator-side flags on the launcher; they do not change PFlash semantics. A short prompt lane should keep the original defaults.
PFlash compress benefits from a small, fast drafter. The validated choice is Qwen3.5-0.8B in BF16 safetensors with ~5 attention layers. The DFlash drafter for the same target works correctly during decode-after-unpark but is heavier than ideal for compress.
Practical guidance:
- Use Qwen3.5-0.8B BF16 for
compress(PFlash side). - Reuse the larger DFlash drafter for
decodeafter unpark (DFlash side).
This dual-drafter setup avoids loading two large drafters simultaneously on a 24 GB GPU.
Reproducible comparison vs Ollama native /api/chat on the same
64K unique-prompt summary task, RTX 6000 Ada sm_89,
Qwen3.6-27B-Q4_K_M, FA_WINDOW=0. Drafter setup: Qwen3.5-0.8B BF16
GGUF for the PFlash compress path (see "Drafter selection" above);
the larger DFlash drafter on the luce daemon side ran as FP16
safetensors during decode-after-unpark on this run. Feel free to
substitute either drafter side with the format you have on disk —
the speedup comes from the compress path, not the dtype:
| Backend | TTFT @ 64K |
|---|---|
Ollama native, qwen3.6:27b-q4_K_M |
68.614 s |
| Lucebox + PFlash compress, OpenAI-compat lane | 23.748 s |
Speedup ~2.89x measured 2026-05-07. Methodology: warm model, single
unique prompt (no prefix-cache reuse), temperature=0,
max_tokens=160. Ollama via native /api/chat (the OpenAI
endpoint of Ollama returned content="" and the response in
message.reasoning, so the comparison was done on the native
endpoint).
The speedup comes from the in-process compress path published in #70; this section only documents how to reproduce it cleanly without prompt caching artefacts.
@software{luce_pflash_2026,
title = {Luce PFlash: speculative prefill compression for long-context spec decode on consumer GPUs},
author = {Lucebox},
url = {https://github.com/Luce-Org/lucebox-hub/tree/main/pflash},
year = {2026}
}
@article{spec_prefill_xfamily_2026,
title = {Cross-Family Speculative Prefill},
author = {Liu and others},
journal = {arXiv:2603.02631},
year = {2026}
}
@article{flashprefill_2026,
title = {FlashPrefill: Block-Sparse Attention for Long-Context Prefill},
author = {Fan and others},
journal = {arXiv:2603.06199},
year = {2026}
}Apache 2.0 · Lucebox · Discord
Inspired by Jingyu6/speculative_prefill, qhfan/FlashPrefill, z-lab/DFlash.