Skip to content

add fireredasr2-aed stt model - #559

Merged
lucasnewman merged 4 commits into
Blaizzy:mainfrom
mm65x:add-fireredasr2-stt
Mar 12, 2026
Merged

lucasnewman merged 4 commits into
Blaizzy:mainfrom
mm65x:add-fireredasr2-stt

Conversation

@mm65x

@mm65x mm65x commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

Context

FireRedASR2-AED (https://github.com/FireRedTeam/FireRedASR2S) from Xiaohongshu/AntGroup. conformer encoder + transformer decoder model for Chinese and English ASR. ~1.18B params, strong on Mandarin (including 20+ dialects/accents), English, and code-switching.

Description

adds FireRedASR2-AED to the stt pipeline. the model uses a 16 layer conformer encoder with relative positional attention (Transformer-XL style with pos_bias_u/v) and a 16 layer transformer decoder with beam search decoding.

the upstream repo distributes weights as a single model.pth.tar with an argparse namespace for config, so this includes a conversion script that extracts weights to safetensors, converts the kaldi CMVN stats to json, and copies the tokenizer files.

the tokenizer is a hybrid system - Chinese characters are treated as individual tokens while English text goes through SentencePiece BPE (~1000 merges).

Changes in the codebase

  • mlx_audio/stt/models/fireredasr2/fireredasr2.py - model implementation (fbank frontend, CMVN, Conv2d subsampling, conformer encoder, transformer decoder, beam search, generate)
  • mlx_audio/stt/models/fireredasr2/config.py - config dataclasses
  • mlx_audio/stt/models/fireredasr2/convert.py - pth.tar → safetensors converter
  • mlx_audio/stt/models/fireredasr2/README.md - setup + usage
  • mlx_audio/stt/models/__init__.py, mlx_audio/stt/utils.py - registration

Changes outside the codebase

none.

Additional information

  • verified end-to-end against the PyTorch reference implementation on Chinese, English, and out-of-distribution (German) audio - transcriptions match exactly
  • encoder intermediate outputs compared numerically: mean abs diff of 0.005 across 16 conformer blocks (expected for float32 cross-framework differences)
  • beam search works for beam_size 1, 3, 5, 10
  • users need to download weights from ModelScope and run the conversion script (instructions in the README)

Checklist

@lucasnewman

lucasnewman commented Mar 10, 2026 •

Copy link
Copy Markdown
Collaborator

@mm65x Can you just upload the converted model to huggingface? Then we don't need the kaldiio dependency.

@mm65x

mm65x commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

@lucasnewman uploaded the converted model here: https://huggingface.co/mlx-community/FireRedASR2-AED-mlx

i've also removed convert.py, updated the README to use mlx-community/FireRedASR2-AED-mlx, and fixed the test file formatting in the latest commit. let me know if there's anything else needed here.

@lucasnewman

Copy link
Copy Markdown
Collaborator

Awesome, thanks!

@lucasnewman
lucasnewman merged commit 907f143 into Blaizzy:main Mar 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support FireRedASR2S

3 participants