Skip to content

Commit 81b9030

Browse files
committed
Reduce some magical numbers
1 parent d7733ca commit 81b9030

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

tools/checkpoint/llava_onevision/convert_to_fs_qwen2.5_1.5b.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -224,9 +224,9 @@ def convert(input_path, output_path, tensor_parallel_size, use_te):
224224

225225
# Process vision tower
226226
# Indices from mapping pytorch multihead attention to megatron.
227-
kv_channels = 72
228227
hidden_dim = 1152
229228
num_heads = 16
229+
kv_channels = hidden_dim // num_heads
230230
indices = []
231231
for i in range(num_heads):
232232
lb = i * kv_channels

0 commit comments

Comments
 (0)