Problem
check_fit adds the preset's flat overheadGiB to the weights and compares with the GPU's memory. vLLM's own admission is different: after the weights and activations it needs max_model_len × KV bytes per token for at least one sequence, and refuses to start otherwise. On one 48 GB card, gemma-4-31b (31.7 GiB of FP8 weights, --max-model-len=65536) was judged fits, 44.0 / 48.0 GiB and then crash-looped: 12.04 GiB KV cache is needed … 7.25 GiB available … estimated maximum model length is 8624. A preset can pass the fit check and never serve; reservedBytes also read 0 on a node with a 20 GiB model loaded.
Proposed solution
Derive the KV bytes per token from the model's config.json (layers × KV heads × head dim × 2 × dtype bytes; MLA and sliding-window layers per the architecture) and the preset's --max-model-len and --kv-cache-dtype, and require weights + activations + max_model_len × KV/token ≤ gpu_memory_utilization × memory for one sequence; report the estimated maximum model length in the verdict the way vLLM does. Count a loaded model's memory in reservedBytes.
Acceptance criteria
Problem
check_fitadds the preset's flatoverheadGiBto the weights and compares with the GPU's memory. vLLM's own admission is different: after the weights and activations it needsmax_model_len × KV bytes per tokenfor at least one sequence, and refuses to start otherwise. On one 48 GB card,gemma-4-31b(31.7 GiB of FP8 weights,--max-model-len=65536) was judgedfits, 44.0 / 48.0 GiBand then crash-looped:12.04 GiB KV cache is needed … 7.25 GiB available … estimated maximum model length is 8624. A preset can pass the fit check and never serve;reservedBytesalso read 0 on a node with a 20 GiB model loaded.Proposed solution
Derive the KV bytes per token from the model's
config.json(layers × KV heads × head dim × 2 × dtype bytes; MLA and sliding-window layers per the architecture) and the preset's--max-model-lenand--kv-cache-dtype, and requireweights + activations + max_model_len × KV/token ≤ gpu_memory_utilization × memoryfor one sequence; report the estimated maximum model length in the verdict the way vLLM does. Count a loaded model's memory inreservedBytes.Acceptance criteria
check_fitforgemma-4-31bat 65536 on a 48 GB card answers does-not-fit with the maximum model length that would; at 32768 it fits.reservedBytesreflects a served model on the node.