feat: add Higgs Audio v2 — 3B Llama-backed TTS with voice cloning

[Kairos-a]

Closes #1.

This PR adds Higgs Audio v2, Boson AI's 3B Llama-3.2-backed TTS with multi-codebook acoustic tokens and delay-pattern streaming. The port reuses the in-tree HiggsAudioTokenizer (added 2026-04-14 for the OmniVoice PR) for the codec path and ships a complete serve-engine equivalent for voice cloning.

Benchmarks (M5 Max, warm cache, long-prompt RTF)

variant	RTF	weights size	source
bf16	0.60×	6.8 GB	bosonai/higgs-audio-v2-generation-3B-base (authoritative)
q8	0.36×	6.18 GB	mlx-community/higgs-audio-v2-3B-mlx-q8
q6	0.33×	4.75 GB	mlx-community/higgs-audio-v2-3B-mlx-q6

Real-time voice cloning on Apple Silicon. bf16 loads directly from the authoritative bosonai upload — no redundant mlx-community re-host. q8 and q6 are MLX-specific selectively-quantized variants.

What's added

Model — mlx_audio/tts/models/higgs_audio/

• HiggsAudioConfig / HiggsTextConfig — from-dict constructor for the bosonai config.json.
• HiggsAudioModel — Llama-3.2-3B backbone with Higgs-specific dual-FFN audio layers (shared attention, split pre-attn-norm + post-attn-norm + MLP between text and audio paths, routed by audio_out_mask). HiggsAudioDecoderProjector with separate text and audio LM heads. Full generate() implementing the AUDIO_INIT + delay-pattern ramp-in + EOS ramp-out state machine.
• generation.py — build_delay_pattern_mask, revert_delay_pattern, apply_delay_pattern, lookup_audio_embedding, greedy_sample_audio, sample_audio (temperature + top-p + top-k via Gumbel-max).

Serving — serve.py

• HiggsAudioServer.from_pretrained() — loads model + codec + tokenizer in one call.
• generate() — ChatML prompt assembly (voice-clone mode when reference_audio_path is provided, smart-voice mode otherwise), reference audio encoded via the in-tree HiggsAudioTokenizer, inputs_embeds stitched with the audio_out_mask routing through the dual-FFN audio path.
• generate_stream() — yields PCM chunks at chunk_ms boundaries for Pipecat / streaming consumers. Quality-preserving (per-chunk identical to non-streaming). Mid-generation emission with overlap-add is noted as follow-up work.
• RAS (repetition-avoidance sampling) on by default (Higgs: ras_win_len=7, ras_max_repeat=2).

Docs — docs/models/tts/higgs_audio.md covers basic usage, voice cloning, streaming, quantization (including the class_predicate pattern that protects the audio head from quantization), sampling controls, and the generation state machine notes.

Tests — mlx_audio/tts/tests/test_higgs_audio.py — 9 unit tests (0.11s on M5 Max) covering delay-pattern shape + round-trip invariants, audio embedding lookup, sampling equivalence at T=0, and model forward on a tiny synthetic config.

Example — examples/higgs_audio_clone_demo.py — CLI demo matching the omnivoice_clone_demo.py convention.

The non-obvious piece — AUDIO_INIT

The generation state machine is the gotcha for this port. The first audio frame must not be sampled from audio_logits at the <|audio_out_bos|> text position — those logits have huge bias toward audio_stream_eos_id because the model was trained with the <|AUDIO_OUT|> placeholder-substitution convention, not direct audio prediction. Instead, a synthetic all-audio_stream_bos_id frame is force-injected (AUDIO_INIT), then the K-frame delay-pattern ramp-in begins. Codebook i starts emitting at frame i; rest stay BOS. On any codebook emitting EOS, a K-frame ramp-out forces trailing codebooks to EOS before termination.

Skipping this produces a stuck pitch (first-frame EOS on half the codebooks → rest is garbage). I've documented this in docs/models/tts/higgs_audio.md so future ports/maintainers don't rediscover it.

Checkpoints

• bf16 — use bosonai/higgs-audio-v2-generation-3B-base directly (authoritative, no re-host needed)
• q8 — mlx-community/higgs-audio-v2-3B-mlx-q8
• q6 — mlx-community/higgs-audio-v2-3B-mlx-q6
• codec — mlx-community/higgs-audio-v2-tokenizer (upstream bosonai repacked into the audio_tokenizer/ subdir layout that mlx-audio's HiggsAudioTokenizer.from_pretrained expects)

Quantized variants carry a quantization block in config.json so HiggsAudioServer.from_pretrained auto-quantizes the skeleton before loading weights.

Quantization notes

MLX native nn.quantize works on the Llama backbone. The docs recommend protecting audio_codebook_embeddings and audio_decoder_proj.audio_lm_head from quantization via class_predicate — quantizing those specific layers introduces voice-character artifacts (pitch-register drift at q6 without protection, trajectory instability at q4).

q4 is deferred from this PR. Rounding noise at 4-bit pushes per-codebook logits close to the sampling decision threshold, so specific Gumbel draws produce clean output while others diverge. The pattern is seed-dependent, not fixable by temperature tuning alone. Follow-up candidates: AWQ-style post-training calibration, smaller group_size, or active retry-on-spiral detection.

Testing

python -m unittest mlx_audio.tts.tests.test_higgs_audio -v
# 9 tests pass in 0.11s

python examples/higgs_audio_clone_demo.py \
    --ref_audio path/to/reference.wav \
    --ref_text "Transcript of the reference clip." \
    --text "Hello from Higgs Audio on MLX."

Credits

Port built on top of the in-tree HiggsAudioTokenizer from the OmniVoice PR (#630). Model code reimplemented from bosonai/higgs-audio against the Llama-3.2-3B architecture in mlx-lm.

[lucasnewman]

@Kairos-a Thanks for the contribution! Overall this looks pretty good -- see a couple of comments around how it can better integrate into the existing codebase.

[lucasnewman]

@Kairos-a Can you add a README.md in the model directory (see the other models for examples) and then we can merge? Thanks!

[lucasnewman]

Thanks!