🚨 EP: fix EP router contract for many models + honor FP8 scale format#46818
🚨 EP: fix EP router contract for many models + honor FP8 scale format#46818IlyasMoutawwakil wants to merge 15 commits into
Conversation
IlyasMoutawwakil
changed the title
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
IlyasMoutawwakil
changed the title
| return Fp8Quantize(self.hf_quantizer) | ||
|
|
||
|
|
||
| class Fp8DecodeScale(ConversionOps): |
There was a problem hiding this comment.
any ideas as to why this part was dropped ?
There was a problem hiding this comment.
because i added support for ue8m0 scales in finegrained-fp8 v3, this was needed for minimax m3 with the v2, but not anymore, it also wastes memory
There was a problem hiding this comment.
ue8m0 scales are a bit messy, some store them in the correct torch dtype, some store them in uint8, and some even store them in fp32 for no special reason 😭 i'm trying to tighten the contract and honor the config all the times because supporting all the on-disk variations would be more complicated
There was a problem hiding this comment.
okay ! Just to be sure, if we remove it now, it would not break existing checkpoints that are in mxpf8 format right ?
There was a problem hiding this comment.
no they will work fine, even better because I just noticed that the fp32 scales are even avoiding the optimized mxfp8 path in https://github.com/huggingface/kernels-community/blob/aeb8ef0e09a132a6583c0a4c8b1096292922b54a/finegrained-fp8/torch-ext/finegrained_fp8/utils.py#L64 I also ran minimax m3 integration tests on the b200
| intermediate_size = config.moe_intermediate_size * config.n_shared_experts | ||
| self.shared_experts = DeepseekOcr2TextMLP(config=config, intermediate_size=intermediate_size) | ||
| self.n_routed_experts = config.n_routed_experts | ||
| self.num_experts = config.n_routed_experts |
There was a problem hiding this comment.
yeh i guess we can drop n_routed_experts, removing it
There was a problem hiding this comment.
hmm so it seems to cascade into many models
There was a problem hiding this comment.
maybe better to add _skip_if_ep_not_supported here instead of within test_ep_*?
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Only checked the modeling parts re modular and the models themself. It is slightly breaking technically because we move parts around modules so let's add 🚨
Generally aligned with this, just a bit unsure about the minimax m3 change - are we keeping everything as is without dequanting and then only convertin after all conversions? Not sure I can follow there 100%
| self.routed_scaling_factor = config.routed_scaling_factor | ||
| self.register_buffer("e_score_correction_bias", torch.zeros(self.num_experts)) | ||
| self.n_routed_experts = config.n_routed_experts + (config.zero_expert_num or 0) | ||
| self.register_buffer("e_score_correction_bias", torch.zeros(self.n_routed_experts)) |
There was a problem hiding this comment.
need to check the modular file here ig
IlyasMoutawwakil
changed the title
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Just some quick comments, ping me when it's ready for another review. Seems like some comments were resolved but not addressed?
|
|
||
|
|
||
| class Lfm2MoeSparseMoeBlock(nn.Module): | ||
| class Lfm2MoeTopkRouter(nn.Module): |
There was a problem hiding this comment.
You dont have to use the deepseek based ones, e.g. just checked qwen3 moe is similar (just missing the scaling factor and expert bias)
def __init__(self, config):
super().__init__(self, config)
self.use_experts_bias = config.user_expert_bias
self.routed_scaling_factor = config.routed_scaling_factor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: afmoe, cohere2_moe, deepseek_ocr2, deepseek_v2, deepseek_v3, deepseek_v32, dots1, ernie4_5_moe, ernie4_5_vl_moe, exaone_moe, flex_olmo, glm4_moe, glm4_moe_lite, glm4v_moe, glm_moe_dsa, hunyuan_v1_moe |
|
CI Dashboard: View test results in Grafana |
4 participants
What does this PR do?
Fixes # (issue)
Code Agent Policy
The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.
PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.
This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read
CONTRIBUTING.md.Before submitting
Pull Request checks?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.