Skip to content

check_fit passes configurations vLLM refuses: the overhead must include max_model_len × KV bytes per token #149

Description

@teemow

Problem

check_fit adds the preset's flat overheadGiB to the weights and compares with the GPU's memory. vLLM's own admission is different: after the weights and activations it needs max_model_len × KV bytes per token for at least one sequence, and refuses to start otherwise. On one 48 GB card, gemma-4-31b (31.7 GiB of FP8 weights, --max-model-len=65536) was judged fits, 44.0 / 48.0 GiB and then crash-looped: 12.04 GiB KV cache is needed … 7.25 GiB available … estimated maximum model length is 8624. A preset can pass the fit check and never serve; reservedBytes also read 0 on a node with a 20 GiB model loaded.

Proposed solution

Derive the KV bytes per token from the model's config.json (layers × KV heads × head dim × 2 × dtype bytes; MLA and sliding-window layers per the architecture) and the preset's --max-model-len and --kv-cache-dtype, and require weights + activations + max_model_len × KV/token ≤ gpu_memory_utilization × memory for one sequence; report the estimated maximum model length in the verdict the way vLLM does. Count a loaded model's memory in reservedBytes.

Acceptance criteria

  • check_fit for gemma-4-31b at 65536 on a 48 GB card answers does-not-fit with the maximum model length that would; at 32768 it fits.
  • reservedBytes reflects a served model on the node.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions