feat(lora) Add lora bgmv_shrink & bgmv_expand kernels

[chaojun-zhang]

Purpose

This PR primarily migrates the LoRA-related ops(bgmv_shrink & bgmv_expand .etc.) from the original IPEX library and focuses on implementing minor optimizations, such as eliminating unnecessary overhead related to unrolling and replacing macros with templates.

Test Plan

pytest tests/test_lora_ops.py

Test Result

BenchMark:

list_bench example:
python3 benchmark/benchmark_lora.py list_bench --arg-pool-size 32 --batch-sizes 1 16 32 --dtype torch.float16 --hidden-sizes 2048 --lora-ranks 16 --num-loras 1 4 --op-types bgmv_shrink bgmv_expand bgmv_expand_slice --seq-lengths 1 --sort-by-lora-id 1

LoRA Benchmark Performance Comparison (float16) - Optimized Operations

Configuration Parameters	Operation Type	ipex (μs)	xpu_kernel (μs)	Performance Improvement
bs=1, sl=1, m=1, k=2048, n=16, num_loras=1	BGMV_SHRINK (f16xf16=>f32)	9.9	7.0	+29.3%
bs=1, sl=1, m=1, k=16, n=2048, num_loras=1	BGMV_EXPAND (f32xf16=>f16)	11.2	8.1	+27.7%
bs=1, sl=1, m=1, k=16, n=2048, num_loras=1	BGMV_EXPAND_SLICE (2 slices)	22.8	16.6	+27.2%
bs=1, sl=1, m=1, k=16, n=2048, num_loras=1	BGMV_EXPAND_SLICE (3 slices)	34.0	24.2	+28.8%
bs=1, sl=1, m=1, k=2048, n=16, num_loras=4	BGMV_SHRINK (f16xf16=>f32)	10.0	7.0	+30.0%
bs=1, sl=1, m=1, k=16, n=2048, num_loras=4	BGMV_EXPAND (f32xf16=>f16)	11.3	8.1	+28.3%
bs=1, sl=1, m=1, k=16, n=2048, num_loras=4	BGMV_EXPAND_SLICE (2 slices)	23.0	16.8	+27.0%
bs=1, sl=1, m=1, k=16, n=2048, num_loras=4	BGMV_EXPAND_SLICE (3 slices)	34.0	24.5	+27.9%
bs=16, sl=1, m=16, k=2048, n=16, num_loras=1	BGMV_SHRINK (f16xf16=>f32)	10.0	7.0	+30.0%
bs=16, sl=1, m=16, k=16, n=2048, num_loras=1	BGMV_EXPAND (f32xf16=>f16)	11.4	8.2	+28.1%
bs=16, sl=1, m=16, k=16, n=2048, num_loras=1	BGMV_EXPAND_SLICE (2 slices)	22.9	16.6	+27.5%
bs=16, sl=1, m=16, k=16, n=2048, num_loras=1	BGMV_EXPAND_SLICE (3 slices)	34.0	24.5	+27.9%
bs=32, sl=1, m=32, k=2048, n=16, num_loras=1	BGMV_SHRINK (f16xf16=>f32)	9.9	7.0	+29.3%
bs=32, sl=1, m=32, k=16, n=2048, num_loras=1	BGMV_EXPAND (f32xf16=>f16)	11.4	8.9	+21.9%
bs=32, sl=1, m=32, k=16, n=2048, num_loras=1	BGMV_EXPAND_SLICE (2 slices)	22.9	18.4	+19.7%
bs=32, sl=1, m=32, k=16, n=2048, num_loras=1	BGMV_EXPAND_SLICE (3 slices)	34.0	27.6	+18.8%

[rogerxfeng8]

@chaojun-zhang can you pls update the pr description?

[jikunshang]

can you rebase?

[chaojun-zhang]

can you rebase?

updated

[jikunshang]

why named is_half here? I understand it's copied from ipex.

[chaojun-zhang]

rename to is_lowp?

[jikunshang]

maybe is_narrow_float or is_low_bit_float

[chaojun-zhang]

is_narrow_float

[jikunshang]

Overall LGTM.