[Question] Supporting RL training of other model families

[MikaStars39]

Will AReaL support RL training of other model families e.g., GPT-OSS, Kimi K2, GLM4.5 ? I've seen there was a roadmap of 2025 Q3 #257.

[garrett4wade]

I think it would be straightforward to add a model in the archon backend, which is motivated by torchtitan, with the help AI.

Specifically, you can clone the torchtitan code, start opencode, and prompt "According to the implementation in @torchtitan, use the /add-archon-model skill to add a new GPT-OSS model in the archon backend. You should take special care of precision alignment." This is actually how we do it. You can make an attempt and create a PR if you succeed. The AI model used for programming does not need to be very smart - using the DeepSeek V3 would suffice.

We don't have many GPUs at this time - even if we have implemented new models, we don't have the resource to test it. Our internal plan is first adding LoRA support within archon, and then run RL for large models with LoRA, which is the timing when we can indeed support and test more new models. Thank you for your understanding. :)

[MikaStars39]

Hi @garrett4wade, thanks for the reply!
I'm currently scaling experiments related to LoRA RL (link: https://arxiv.org/abs/2512.23165). I'd love to help test Archon’s LoRA support and handle CI testing for any new models you're integrating.

[garrett4wade]

Hi @MikaStars39 , thank you for your interests!

We are currently mid-way through the implementation on the fw/archon-lora branch. To get started, check out the LoRA implementation plan to see where we're headed. We've separate it into several phases to implement progressively. We'll leave the branch unchanged.

The most helpful next step would be reviewing the diff between that branch and main, then helping us finalize Phase 1 with a clean PR. Make sure to peek at our AI-assisted dev docs and the contributing guide—they should make the process much smoother!

cc our human expert on archon engine @rchardx

[NJX-njx]

I'd like to work on this! I'll port the Phase 1 (core LoRA infrastructure) and Phase 2 (checkpointing & PEFT conversion) from the w/archon-lora branch to main as a clean PR, following the implementation plan in lora-global-plan.md. I'll include comprehensive tests and ensure PEFT ecosystem compatibility.

[MikaStars39]

Hi @NJX-njx, thanks for the reply! I will test this PR w/ archon backend + Qwen3 LoRA training. Results maybe will updated in a few days.

[MikaStars39]

BTW, I am also interested in some RL-related methods e.g., On-Policy Distillation https://thinkingmachines.ai/blog/on-policy-distillation/, and more RL xxPO algo e.g., CISPO https://arxiv.org/abs/2506.13585. I'd like to add these into AReaL considering they can easily reuse existing GRPO pipeline. I'll raise a PR when I finished. cc @garrett4wade @rchardx

[garrett4wade]

@MikaStars39 That will be great. FYI our fellows from HW have implemented on-policy distillation in PR #964 . We can move on to support other algorithms.

[ZiyiTsang]

I am trying to consider this from the perspective of Megatron backend
Currently, AreaL is using mbridge which is a framework not officially supported by Nvidia, and slow in supporting new model.
If we want to quickly adapt new models from the megaton backend, the Megatron-Bridge by offical may be more user-friendly.
Does Areal have any follow-up plans for this?

[MikaStars39]

I'd like to work on this! I'll port the Phase 1 (core LoRA infrastructure) and Phase 2 (checkpointing & PEFT conversion) from the w/archon-lora branch to main as a clean PR, following the implementation plan in lora-global-plan.md. I'll include comprehensive tests and ensure PEFT ecosystem compatibility.

Hi @NJX-njx , I've fixed several bugs in LoRA-Archon training in this pr #1015 . Now it can successfully enter the training loop and checkpoints can be saved and loaded correctly from the training / infer backend. I will continue to test the stability of lora training on an RLVR exp w/ DAPO-17k dataset.

[gursimar]

@ZiyiTsang @garrett4wade I'm curious regarding Megatron-Bridge as well. Since it is the officially supported repo, does AReaL has any plan/ need to migrate from mbridge to MegatronBridge? Megatron Bridge has more model support and also LORA support. Are you aware of anyone working/ planning to work on this?

[garrett4wade]

@gursimar I did some preliminary study before. Integrating Megatron-Bridge requires rewriting the current save-load code that was heavily optimized towards mbridge. We indeed want to integrating more models from MegatronBridge, but at least it requires some rewriting job. This is not our major focus right now.