Skip to content

Conversation

jlamypoirier
Copy link
Collaborator

@jlamypoirier jlamypoirier commented Aug 29, 2025

✨ Description

Replace inheritance with composition for MLP and mixer configs

Also:

  • Extract MoE config into a separate dynamic MLP type.
  • Remove most ad-hoc initialization arguments
  • Merge mlp_lr_scale and router_lr_scale into a single lr_scale
  • Add set_defaults for defaults that depend on the hidden size

Todo (next prs):

  • Add back fine-grained lr_scale
  • Add back fine-grained bias enabling config (quen2 and dream disabled).
  • Add back fine-grained initialization configuration.
  • Rework SSM conversion (disabled).

@jlamypoirier jlamypoirier marked this pull request as ready for review August 29, 2025 20:46
@jlamypoirier jlamypoirier marked this pull request as draft August 29, 2025 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant