Skip to content

CUDA: vectorize FP16 attention input preprocessing - #2436

Open
john-sp wants to merge 1 commit into
LeelaChessZero:masterfrom
john-sp:cuda-pr2-preprocess-fp16x8
Open

CUDA: vectorize FP16 attention input preprocessing#2436
john-sp wants to merge 1 commit into
LeelaChessZero:masterfrom
john-sp:cuda-pr2-preprocess-fp16x8

Conversation

@john-sp

@john-sp john-sp commented Aug 1, 2026

Copy link
Copy Markdown
Member
  • Add an eight-elements-per-thread FP16 path for PE-dense attention-body preprocessing.
  • Use aligned 128-bit loads and stores where channel boundaries permit.
  • Retain the scalar implementation for FP32, non-dense positional encoding, and unsupported alignment.

Dispatch depends only on datatype, layout, and channel alignment; it contains no model, batch-size, device-name, or SM-count checks.

Performance

RTX 3090 Ti, BT4, batch 84:

  • Master median: 23.732 us
  • Optimized median: 14.246 us
  • Kernel reduction: 40.0%
  • Saving: approximately 9.5 us per evaluation

AI Acknowledgement: Written by Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant