[Kernels] Sync and add SwigluMLP + CausalLMLoss

[vasqu]

WIP

Need to fix the upstream init of the kernel - see https://gist.github.com/vasqu/09cc54006faae94460fe6470bc638ed8 (same for rope)

• We had _ instead of - so it pointed to the wrong repo lmao
• We have a few validation strictness issues, e.g.
  • Match signature when fn
  • No init for module
• Torch compile support will also need additional changes in kernels to register can_torch_compile flag if given
• RMS Norm needs investigation why it fails under compile
• We should be able to support training but not compile (rms norm)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

looks nice!

[ArthurZucker]

BTW make sure to put benches on perfs wins /. doc

[vasqu]

Yep, will do 🫡 thinking about memory benchmarks mostly since this is the key behind the new loss kernel from liger

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: llama