segmentation while lora finetuning

[yao-matrix]

🐛 Describe the bug

$ git clone https://github.com/huggingface/peft.git
$ pip install -e .
$ cd ./method_comparison/MetaMathQA
$ touch ./experiments/lora/llama-3.2-3B-rank32/adapter_config.json
$ python run.py -v ./experiments/lora/llama-3.2-3B-rank32

segmentation fault will be reported, while doing the first evaluation of finetuning process, as below:

Versions

pytorch-triton-xpu 3.4.0+gitae324eea
torch 2.8.0.dev20250627+xpu
torchao 0.11.0
torchaudio 2.8.0.dev20250627+xpu
torchdata 0.11.0
torchvision 0.23.0.dev20250627+xpu

[EikanWang]

@zhangxiaoli73 , could you help investigate the seg fault issue?

[EikanWang]

cc @riverliuintel . Since this issue is a seg-fault issue, let's prioritize fixing.

[yao-matrix]

thx all. tried Cherry's comment, and can see both 1 card and multi-card same segfault issue.

To minimize debug waiting time, you can change default_training_params.json like below:

"max_steps": 5,
"eval_steps": 1,

[zhangxiaoli73]

Thanks to @yao-matrix double check, then it is not a problem with communication. @riverliuintel Could you please assign someone to take a look? We need some trace (like unitrace) to figure out the kernel that cause this failure.

[riverliuintel]

@xiaowangintel please prioritize to look at this issue.