Pr 3002 ci branch

[drbh]

This PR re opens #3002 to run CI and add some small changes to align the logic across different hardware

[sywangyi]

Hi, @drbh the change of 7c09eae for rocm and ipex is incorrect, because it's inplace ops ( ops.rotary_embedding/ipex.llm.functional.rotary_embedding) for query/key. but you change query/key tensor in
query = torch.cat([-query[..., rot:], query[..., :rot]], dim=-1)
key = torch.cat([-key[..., rot:], key[..., :rot]], dim=-1)

For Qwen vl rope, I think it's same with llama except sin/cos calulation. that's why I delete forward and torch cat for sin/cos.

[drbh]

Hi @sywangyi you are completely correct! apologies for my misunderstanding the forward and cat should be removed - I've reverted the changed here and if CI passes I'll merge the original PR. Thanks for this fix 🙏

[drbh]

closing as the original PR was merged #3002