Questions regarding the Qwen2.5-7B-Instruct model

[hh11-A]

Hello, the paper includes results/data for Qwen2.5-7B-Instruct. How can I reproduce/run those results using the project code? I don’t see any configuration related to Qwen2.5 in the codebase. What settings should I add, or what command-line arguments should I use, to achieve the Qwen2.5-7B-Instruct results reported in the paper?

[menik1126]

We have added support for the Qwen model and introduced support for FlashAttention. The newly added model file can be found at: /home/xiongjing/UNComp/uncomp/qwen_model.py. @hh11-A

[hh11-A]

if "llama" in model_path.lower() or "mistral" in model_path.lower() or "qwen" in model_path.lower():
from uncomp.monkeypatch import replace_llama,replace_mistral,replace_qwen
replace_llama(args.method.lower(),manager)
replace_qwen(args.method.lower(),manager)
replace_mistral(args.method.lower(),manager)
func_utils_instance.determine_head_type(manager,device=model.device)

After modifying the code above, it now enters the Qwen model.

bash ./scripts/scripts_longBench/eval.sh
--max_capacity_prompts 384
--attn_implementation eager
--source_path ./results/
--model_path /data/Qwen2.5-7B-Instruct
--eval_batch_size 1
--method uncomp_stage
--name ./output/Qwen2.5-7B-Instruct
--gpu_id 0
--fp16 1
--seed 43
--logger_pattern info
--port 1236
After applying the above modifications, the code is able to enter the Qwen model. However, it still crashes when running with the above command. I have fixed many bugs, but it is still not able to run successfully. Could you please advise how to make it work? Thank you very much for your help. @menik1126

[menik1126]

Please contact the author via email and include your error message for further handling. @hh11-A

[hh11-A]

This is the main source of bugs.

There is also a small issue with the monkeypatch code. The mistral_attn_forward_Uncomp and MistralDecoderLayer_forward functions are swapped, and qwen cannot be found in init—it should be qwen2 instead. After fixing this, I encountered the bug mentioned above.

Author, were you able to run it successfully, or is it an issue with my configuration? I managed to run llama2-8/13 3-8 without problems, but encountered issues specifically with the qwen model.
If there's something wrong with my setup, I'd appreciate your guidance. Thank you. @menik1126

[menik1126]

This issue has been resolved. Please test again and compare the results with those in the main text. We ran the experiments on an AMD MI210 machine, so the accuracy may be slightly lower than that on an NVIDIA machine. @hh11-A

[hh11-A]

Hello, sorry to bother you again.
I’m still encountering some runtime issues on Qwen2.5-7B.
When using --method uncomp, it runs successfully.
However, it fails when using --method uncomp_stage.

The bug is as follows:
line 912, in qwen_attn_forward_Uncomp
！
Could you help fix this? Also, would it run properly on newer models such as Qwen3/Qwen3.5 or Llama 3.1? I tried it myself, but it failed.
/cc @menik1126