Merge branch 'main' into fix/whisper-empty-non-speech-tokens

Author: saichowdary007

.github/workflows/tests.yml

@@ -43,6 +43,11 @@ jobs:
       - name: Install all deps
         run: pip install -e ".[all,dev]"
 
+      - name: Install ffmpeg for server audio tests
+        run: |
+          brew install ffmpeg
+          ffmpeg -version
+
       - name: Test DSP imports without TTS/STT
         run: |
           python -c "from mlx_audio.dsp import stft, mel_filters; print('DSP OK')"

CONTRIBUTIONS.md

@@ -29,3 +29,10 @@ This file acknowledges the original authors and contributors of models ported to
 - **Copyright**: NVIDIA Corporation
 - **License**: [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
 - **MLX Port**: [@ARahim3](https://github.com/ARahim3)
+
+## rumik-oss 1 (Text-to-Speech)
+
+- **Original**: [rumik-ai/rumik-oss-1](https://huggingface.co/rumik-ai/rumik-oss-1)
+- **Copyright**: rumik.ai
+- **License**: [CC-BY-NC 4.0 with Cohere Labs acceptable-use addendum](https://cohere.com/cohere-labs-cc-by-nc-license) (weights); Mimi codec CC-BY-4.0
+- **MLX Port**: Suryansh Shakya ([@nullHawk](https://github.com/nullHawk))

README.md

@@ -132,6 +132,7 @@ for result in model.generate(
 | **Ming Omni TTS (Dense)** | Lightweight dense Ming Omni variant for voice cloning and style control | EN, ZH | [mlx-community/Ming-omni-tts-0.5B-bf16](https://huggingface.co/mlx-community/Ming-omni-tts-0.5B-bf16) |
 | **KugelAudio** | SOTA 7B AR+Diffusion TTS for European languages | EN, DE, FR, ES, IT, PT, NL, PL, RU, UK, + 14 more | [kugelaudio/kugelaudio-0-open](https://huggingface.co/kugelaudio/kugelaudio-0-open) |
 | **Voxtral TTS** | Mistral's 4B multilingual TTS (20 voices, 9 languages) | EN, FR, ES, DE, IT, PT, NL, AR, HI | [mlx-community/Voxtral-4B-TTS-2603-mlx-bf16](https://huggingface.co/mlx-community/Voxtral-4B-TTS-2603-mlx-bf16) |
+| **rumik-oss 1** | 3B expressive multilingual Indic TTS with 22-language support, description-conditioned delivery and inline vocalizations | 22 Indic languages + EN | [rumik-ai/rumik-oss-1](https://huggingface.co/rumik-ai/rumik-oss-1), [8bit](https://huggingface.co/rumik-ai/rumik-oss-1-mlx-8bit), [4bit](https://huggingface.co/rumik-ai/rumik-oss-1-mlx-4bit) |
 | **VoxCPM2** | 2B tokenizer-free TTS with 48kHz output, voice design, voice cloning, and continuation | 30 languages | [bf16](https://huggingface.co/mlx-community/VoxCPM2-bf16), [8bit](https://huggingface.co/mlx-community/VoxCPM2-8bit), [4bit](https://huggingface.co/mlx-community/VoxCPM2-4bit) |
 | **LongCat-AudioDiT** | SOTA diffusion TTS in waveform latent space with voice cloning | ZH, EN | [mlx-community/LongCat-AudioDiT-1B-bf16](https://huggingface.co/mlx-community/LongCat-AudioDiT-1B-bf16) |
 | **MeloTTS** | Lightweight VITS2-based TTS with streaming | EN (more coming) | [mlx-community/MeloTTS-English-MLX](https://huggingface.co/mlx-community/MeloTTS-English-MLX) |
@@ -178,7 +179,9 @@ See the model READMEs for API details, streaming examples, and conversion steps.
 | Model | Description | Use Case | Repo |
 |-------|-------------|----------|------|
 | **SAM-Audio** | Text-guided source separation | Extract specific sounds | [mlx-community/sam-audio-large](https://huggingface.co/mlx-community/sam-audio-large) |
+| [**DialogueSidon**](mlx_audio/sts/models/dialogue_sidon/README.md) | Two-speaker separation and restoration | Separate dialogue into speaker tracks | [mlx-community/DialogueSidon](https://huggingface.co/mlx-community/DialogueSidon) (FP32), [mlx-community/DialogueSidon-bf16](https://huggingface.co/mlx-community/DialogueSidon-bf16) (BF16) |
 | **Liquid2.5-Audio*** | Speech-to-Speech, Text-to-Speech and Speech-to-Text | Speech interactions | [mlx-community/LFM2.5-Audio-1.5B-8bit](https://huggingface.co/mlx-community/LFM2.5-Audio-1.5B-8bit) |
+| **MiMo-Audio** | English/Chinese TTS, ASR, audio understanding and dialogue; Base few-shot speech tasks | Speech interactions and audio completion | [Instruct](https://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Instruct), [Base](https://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Base), [audio tokenizer](https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer), [guide](mlx_audio/sts/models/mimo_audio/README.md) |
 | **MossFormer2 SE** | Speech enhancement | Noise removal | [starkdmi/MossFormer2_SE_48K_MLX](https://huggingface.co/starkdmi/MossFormer2_SE_48K_MLX) |
 | **DeepFilterNet (1/2/3)** | Speech enhancement | Noise suppression | [mlx-community/DeepFilterNet-mlx](https://huggingface.co/mlx-community/DeepFilterNet-mlx) |
 | **NemotronLabs VoiceChat** | Full-duplex speech-to-speech with streaming transcription and function calling | Real-time voice conversation | [mlx-community/NemotronLabs-VoiceChat-11B-4bit](https://huggingface.co/mlx-community/NemotronLabs-VoiceChat-11B-4bit) |

docs/guides/nemotron-live-input.md

@@ -0,0 +1,48 @@
+# Nemotron live input
+
+Nemotron implements the shared [`StreamingSession`](streaming-stt.md) protocol,
+also used by Voxtral Realtime. Sessions use greedy
+decoding (`temperature=0`), mono float32 PCM at `session.input_sample_rate`, and
+independent frontend, encoder and RNNT state.
+
+```python
+session = model.create_streaming_session()
+session.feed(samples)
+print("".join(session.step(max_decode_tokens=8)), end="")
+session.close()
+while not session.done:
+    print("".join(session.step(max_decode_tokens=8)), end="")
+```
+
+Call `step` regularly from one decoder thread. Its budget counts joint evaluations,
+including blanks and special tokens. Each step ingests at most one native audio
+chunk and drains existing encoded frames before ingesting more. Packet boundaries
+do not reset frontend or decoder state. Text is emitted as deltas, not cumulative
+transcripts. No transcript history is retained by the session.
+
+`feed` copies input and rejects non-finite or non-mono data. Pending audio is
+limited to 30 seconds; exceeding that limit raises `BufferError` without dropping
+previously queued audio. This is a pending-input limit, not a session duration
+limit. Producers must pace input and consumers must keep stepping. `close` is
+idempotent and signals input completion, not immediate decoder completion.
+`cancel` discards pending work without emitting a successful final; `reset`
+starts independent state. Both must run with no concurrent decoder step.
+
+The generic server selects models through `/v1/realtime?model=<model-id>` and
+accepts session updates and PCM16 audio append/commit events. Use model-rate PCM
+to avoid conflating session behavior with packet-wise server resampling. This
+change does not add a Nativ route, authentication policy or new server protocol.
+
+Validation covers tiny-model frontend/encoder behavior, real tiny-decoder parity,
+and the actual ASGI WebSocket handler (partial before commit, completion and a
+second turn). A local multilingual checkpoint check used five paced runs each of
+Portuguese and English synthesized speech, with 20 ms / 16 kHz chunks. Text matched
+the existing streaming decoder after trimming outer whitespace. Both paths made
+the same Portuguese recognition error; this is parity evidence, not an accuracy
+benchmark.
+
+For those ten local session runs, first-partial p50/p95 was 1.912/3.433 seconds and
+close-to-final p50/p95 was 0.267/0.676 seconds (maximum 0.921 seconds). Quantiles use
+linear interpolation. These small-sample measurements are not WebSocket latency
+or Nativ/OpenClaw end-to-end proof. Long-session memory-residency measurements,
+broader speech samples and production integration remain outstanding.

docs/guides/quantization.md

@@ -78,6 +78,29 @@ python -m mlx_audio.convert \
     --upload-repo username/Kokoro-82M-4bit
 ```
 
+## MiMo-Audio
+
+Base and Instruct can load the original checkpoints directly. To reduce memory, convert either checkpoint with the standard converter:
+
+```bash
+python -m mlx_audio.convert \
+    --hf-path XiaomiMiMo/MiMo-Audio-7B-Instruct \
+    --mlx-path ./MiMo-Audio-7B-Instruct-4bit \
+    -q --q-bits 4 --q-group-size 64
+```
+
+MiMo's quantization policy covers the global Qwen2 backbone and text head. Acoustic patch transformers, speech embeddings and speech heads retain their floating-point precision. The separate audio tokenizer is not quantized. Consequently, total memory use is larger than a uniformly quantized 7B text model. Both source and converted weights use the same [offline APIs](../models/sts/mimo-audio.md).
+
+To save the standalone tokenizer in MLX convolution layouts:
+
+```bash
+python -m mlx_audio.codec.models.mimo_audio_tokenizer.convert \
+    --hf-path XiaomiMiMo/MiMo-Audio-Tokenizer \
+    --mlx-path ./MiMo-Audio-Tokenizer-mlx
+```
+
+Its converter preserves stored precision by default and accepts `--dtype` and `--revision`. The resulting directory contains weights, config, and a model card. The language-model converter records the separate codec dependency and its pinned revision in `config.json`; use `audio_tokenizer_path` or the CLI's `--audio-tokenizer-path` to choose a local codec. No upload is required for local inference.
+
 ## Conversion Options Reference
 
 | Flag | Description |

docs/guides/streaming-stt.md

@@ -0,0 +1,49 @@
+# Adding a live-input STT model
+
+`mlx_audio.stt.streaming.StreamingSession` is the shared structural protocol
+used by the realtime server. Nemotron and Voxtral Realtime both implement it.
+No base class, registration step, or model-name branch in the server is needed.
+
+Implement `create_streaming_session(...) -> StreamingSession` on the model.
+Keep model-specific construction and options there; the session presents:
+
+- `input_sample_rate: int`: native input rate.
+- `feed(samples: np.ndarray) -> None`: queue mono float32 PCM; no inference.
+- `step(*, max_decode_tokens: int = 4) -> list[str]`: advance on one model
+  executor and return append-only text deltas, possibly an empty list.
+- `close() -> None`: signal end of input, without discarding the pending tail.
+- `done: bool`: true once decoding has finished.
+
+`feed` and `close` may run on a producer thread while one consumer calls `step`.
+Do not modify submitted arrays while queued, or feed after close/completion.
+Keep calling `step` after close until done. An empty result is not completion.
+Models may also finish early (for example EOS or a model-specific token cap).
+The decode budget is not a time limit: model-specific encoder work can still
+make a step expensive. Document input limits and errors in the model guide.
+Reset/cancel are optional implementation extensions, not part of this contract.
+
+```python
+from mlx_audio.stt.streaming import StreamingSession
+
+session: StreamingSession = model.create_streaming_session(temperature=0.0)
+for samples in audio_chunks:
+    if session.done:
+        break
+    session.feed(samples)
+    for delta in session.step():
+        emit(delta)
+session.close()
+while not session.done:
+    for delta in session.step():
+        emit(delta)
+```
+
+This example executes sequentially; a live server schedules the producer and
+consumer independently. Factory options remain model-specific; the existing
+server forwards transcription delay only when the factory declares it.
+
+Add a small model fixture to `mlx_audio/stt/tests/test_streaming_session.py`
+to exercise the same server/session contract, plus model-specific decoder
+parity tests. Runtime `isinstance` checks only member presence; behavioral tests
+are necessary to prove the contract. The shared test uses tiny random weights
+and deterministic token selection, not production checkpoints or downloads.

docs/models/index.md

@@ -30,6 +30,7 @@ Generate natural-sounding speech from text. Multiple models with multilingual su
 | **KittenTTS** | Compact KittenTTS 0.8 models for edge-friendly TTS | EN | [nano](https://huggingface.co/mlx-community/kitten-tts-nano-0.8) / [micro](https://huggingface.co/mlx-community/kitten-tts-micro-0.8) / [mini](https://huggingface.co/mlx-community/kitten-tts-mini-0.8) |
 | **Qwen3-TTS** | Alibaba's multilingual TTS with voice design | ZH, EN, JA, KO, + more | [mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16](https://huggingface.co/mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16) |
 | **Voxtral TTS** | Mistral's 4B multilingual TTS (20 voices, 9 languages) | EN, FR, ES, DE, IT, PT, NL, AR, HI | [mlx-community/Voxtral-4B-TTS-2603-mlx-bf16](https://huggingface.co/mlx-community/Voxtral-4B-TTS-2603-mlx-bf16) |
+| **rumik-oss 1** | 3B expressive multilingual Indic TTS with 22-language support, description-conditioned delivery and inline vocalizations | 22 Indic languages + EN | [rumik-ai/rumik-oss-1](https://huggingface.co/rumik-ai/rumik-oss-1), [8bit](https://huggingface.co/rumik-ai/rumik-oss-1-mlx-8bit), [4bit](https://huggingface.co/rumik-ai/rumik-oss-1-mlx-4bit) |
 | **VoxCPM2** | 2B tokenizer-free TTS with 48kHz output, voice design, voice cloning, and continuation | 30 languages | [bf16](https://huggingface.co/mlx-community/VoxCPM2-bf16), [8bit](https://huggingface.co/mlx-community/VoxCPM2-8bit), [4bit](https://huggingface.co/mlx-community/VoxCPM2-4bit) |
 | **CSM / MisoTTS** | Sesame-style conversational speech models with voice cloning | EN | [mlx-community/csm-1b](https://huggingface.co/mlx-community/csm-1b), [MisoTTS bf16](https://huggingface.co/mlx-community/MisoLabs-MisoTTS-bf16), [MisoTTS 8bit](https://huggingface.co/mlx-community/MisoLabs-MisoTTS-8bit) |
 | **Dia** | Dialogue-focused TTS | EN | [mlx-community/Dia-1.6B-fp16](https://huggingface.co/mlx-community/Dia-1.6B-fp16) |

docs/models/sts/dialogue-sidon.md

@@ -0,0 +1,81 @@
+# DialogueSidon
+
+Separate a recording of two speakers into individual audio tracks on Apple
+Silicon. DialogueSidon also restores degraded speech and outputs two mono WAV
+files at **24 kHz**, with the same duration as the input.
+
+Follow the [MLX Audio installation instructions](../../getting-started/installation.md)
+before running the examples below.
+
+## Supported models
+
+| Precision | Repository ID | Weight size |
+|-----------|---------------|-------------|
+| FP32 | [mlx-community/DialogueSidon](https://huggingface.co/mlx-community/DialogueSidon) | 1.78 GB |
+| BF16 | [mlx-community/DialogueSidon-bf16](https://huggingface.co/mlx-community/DialogueSidon-bf16) | 889 MB |
+
+Either repository ID works in the examples below. Choose BF16 for a smaller
+download and lower weight memory use. You can also load a local model directory.
+
+## Command line
+
+```bash
+python -m mlx_audio.sts.generate \
+    --model mlx-community/DialogueSidon \
+    --audio dialogue.wav --output-path separated.wav \
+    --num-steps 30 --seed 0
+```
+
+This writes `separated_speaker_1.wav` and `separated_speaker_2.wav`.
+
+## Python
+
+```python
+from mlx_audio.sts import load
+from mlx_audio.audio_io import write
+
+model = load("mlx-community/DialogueSidon")
+result = model.separate("dialogue.wav", num_steps=30, seed=0)
+for i, speaker in enumerate(result.speakers, 1):
+    write(f"speaker_{i}.wav", speaker, result.sample_rate)
+```
+
+`result.speakers` contains two tracks with shape `[2, samples]`, and
+`result.sample_rate` is `24000`.
+
+For a NumPy or MLX array, provide its sample rate:
+
+```python
+from mlx_audio.audio_io import read
+
+audio, sample_rate = read("dialogue.wav")
+result = model.separate(audio, sample_rate=sample_rate, seed=0)
+```
+
+Arrays must use `[samples]` or `[samples, channels]` layout. Stereo and other
+multichannel inputs are averaged to mono before separation.
+
+## Options and long recordings
+
+| Python option | CLI option | Default | Purpose |
+|---------------|------------|---------|---------|
+| `num_steps` | `--num-steps` | `30` | Fewer steps run faster but can change output quality. |
+| `seed` | `--seed` | Random | Set a seed to repeat a run with the same input, model, and settings. |
+| `chunk_seconds` | `--chunk-seconds` | `20.0` | Audio processed at once; smaller chunks reduce memory use. |
+| `overlap_seconds` | `--overlap-seconds` | `5.0` | Overlap between chunks, in seconds. Must be less than the chunk duration. |
+
+Long recordings are processed automatically in overlapping chunks. To process
+an entire file at once in Python, use `chunk_seconds=None`; this uses more
+memory.
+
+The model supports exactly two speakers and processes recorded audio offline.
+The tracks are numbered rather than labeled with speaker identities; a speaker
+may switch tracks after a long silence. Because the model also restores speech,
+adding the tracks together may not reproduce the input exactly.
+
+## License and attribution
+
+Model weights use **CC-BY-NC-4.0**. Original model by Wataru Nakata, Yuki Saito,
+Kazuki Yamauchi, Emiru Tsunoo, and Hiroshi Saruwatari (SaruLab).
+See the [original model card](https://huggingface.co/sarulab-speech/DialogueSidon).
+The upstream [Sidon code](https://github.com/sarulab-speech/Sidon) is MIT licensed.