Skip to content

cuda: tune fixed-size softmax launch - #2435

Open
john-sp wants to merge 1 commit into
LeelaChessZero:masterfrom
john-sp:cuda-pr1-softmax64
Open

cuda: tune fixed-size softmax launch#2435
john-sp wants to merge 1 commit into
LeelaChessZero:masterfrom
john-sp:cuda-pr1-softmax64

Conversation

@john-sp

@john-sp john-sp commented Aug 1, 2026

Copy link
Copy Markdown
Member
  • Reduce the optimized 64-element softmax launch from 256 to 128 threads.
  • Compute one exact reciprocal and reuse it for the two outputs handled by each thread.

Performance:

On RTX 3090 Ti at batch 84:

  • Current paired WSL Nsight trace: 18.372 us -> 18.454 us median, effectively neutral.
  • Earlier cleaner isolated traces: 19.828 us -> 19.409 us, approximately 2.1% faster.

AI Acknowledgement: Written by Codex

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tunes the CUDA-optimized softmax path for C == 64 by adjusting the kernel launch configuration and reducing redundant arithmetic in the normalization step.

Changes:

  • Reduce the softmax_opt_64_kernel block size from 256 to 128 threads (4 warps / 4 rows per block).
  • Compute a single reciprocal of the warp-wide sum (inv_sum) and reuse it for both outputs per thread.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants