Long-context prefill (40k+ on Qwen3.5-9B at 131k ctx) blows up memory; chunked prefill is force-disabled in platform plugin

[yaanfpv]

Summary

Long prompts (e.g., 40k tokens) at long max_model_len (e.g., 131k) cause vllm-swift's EngineCore to consume tens of GB of virtual memory on small Macs and either be SIGKILL'd by the OS or compress so heavily that the system becomes unresponsive. The same model + context size handles the same workload comfortably on llama.cpp on the same hardware. The root cause is that vllm-swift force-disables chunked prefill, so the entire prompt is sent to the Swift engine in a single prefill_req call.

Affected versions

• vllm-swift v0.6.0
• vLLM 0.19.1
• macOS 26 (Tahoe), Apple Silicon, 16 GB unified memory

Reproduction

Model: Qwen3.5-9B-VL MLX 4-bit (any Qwen3.5 family will reproduce).

vllm-swift serve <Qwen3.5-9B MLX 4-bit dir> \
  --served-model-name qwen35-9b \
  --max-model-len 131072 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 512 \
  --max-num-seqs 1 \
  --host 127.0.0.1 --port 8080 \
  --trust-remote-code

POST a chat completion with a ~40k-token prompt. Activity Monitor reports the EngineCore process consuming 30 to 60 GB (compressed + swap). Eventually:

ERROR core_client.py:667 Engine core proc EngineCore died unexpectedly, shutting down client.
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

For comparison, llama.cpp on the same hardware with -c 131072 -ub 512 -fa on --cache-type-k q8_0 --cache-type-v turbo4 handles 50k-token prompts at ~150 tok/s prefill with under 8 GB resident.

Root cause

vllm_swift/platform.py lines 112-114 silently override the user's --enable-chunked-prefill flag:

# Disable chunked prefill — Swift engine handles full sequences
if getattr(scheduler_config, "enable_chunked_prefill", False):
    scheduler_config.enable_chunked_prefill = False

The EngineCore config dump confirms enable_chunked_prefill=False even when the user passed --enable-chunked-prefill. The Swift FFI (vsm_engine_prefill_req, vsm_engine_prefill_vlm) only takes a full prompt in one call, so the worker has no way to feed chunks even if vLLM tried.

Without chunked prefill, working memory during prefill scales with the full prompt length: residual stream [seq_len, hidden] plus per-layer activations and gated-delta linear-attention state across all layers stay alive simultaneously. For a 40k prompt on a 9B hybrid model, peak transient memory exceeds available physical RAM on a 16 GB Mac.

Proposed fix scope

Two parts:

1. Swift bridge: add an incremental prefill FFI such as vsm_engine_prefill_chunk(req_id, tokens, is_first, is_last, ...) that accumulates KV / gated-delta state across calls.
2. Worker: slice prompt_token_ids into max_num_batched_tokens-sized chunks and call the new FFI per chunk; remove the platform.py override.

This would let vllm-swift match llama.cpp's -ub 512 behavior and unlock long-context use on Apple Silicon Macs with limited RAM.

Workaround

For now, --max-model-len has to be sized so that the worst-case full-prompt prefill fits in RAM. On 16 GB Macs that is closer to 8k-16k for a 9B model, far below what the model itself supports.

Reported by @yaanfpv.