Skip to content

Moss tts realtime - #505

Closed
voxmenthe wants to merge 7 commits into
Blaizzy:mainfrom
voxmenthe:moss-tts-realtime
Closed

voxmenthe wants to merge 7 commits into
Blaizzy:mainfrom
voxmenthe:moss-tts-realtime

Conversation

@voxmenthe

Copy link
Copy Markdown
Contributor

Realtime Runtime: Session Lifecycle + Streaming + Chunk/Overlap Decoding

This PR adds the dedicated MOSS realtime runtime path, including explicit session lifecycle controls, incremental text-delta streaming, and chunked audio decoding with overlap crossfade.

Scope (Included)

This PR includes:

  • mlx_audio/tts/models/moss_tts_realtime/**
  • mlx_audio/tts/tests/test_moss_tts_realtime_runtime.py
  • tests/tts/models/moss_tts_realtime/test_realtime_regressions.py
  • examples/moss_tts_realtime_multiturn_agent.py
  • examples/moss_tts_realtime_text_deltas.py
  • mlx_audio/tts/models/qwen3_tts/speech_tokenizer.py

Primary behavior in scope:

  • Dedicated realtime inferencer/session runtime (separate from non-realtime flow).
  • Explicit session lifecycle semantics: reset_turn -> text ingestion -> end_text -> drain.
  • Text-delta streaming bridge with incremental emission.
  • Bounded pending-frame buffer (backpressure/fail-fast behavior).
  • Chunked decode with overlap crossfade controls (chunk_frames, overlap_frames).
  • Realtime sampling controls including repetition-window behavior.
  • Realtime examples aligned to the runtime lifecycle contract.

Scope (Intentionally Excluded)

This PR does not include:

  • Non-realtime MOSS runtime behavior (mlx_audio/tts/models/moss_tts/**).
  • Generic TTS CLI gateway refactors in mlx_audio/tts/generate.py.
  • Server/API request schema or endpoint behavior in mlx_audio/server.py.
  • Broad docs/readme expansion outside the realtime examples listed above.

Validation

uv run pytest -q \
  mlx_audio/tts/tests/test_moss_tts_realtime_runtime.py \
  tests/tts/models/moss_tts_realtime/test_realtime_regressions.py

Additional targeted checks run during review:

uv run pytest -q mlx_audio/tts/tests/test_moss_tts_realtime_runtime.py -k \
  'bridge_and_direct_token_paths_produce_equal_chunk_lengths or \
   overlap_crossfade_non_final_decode_path_is_mlx_safe or \
   overlap_crossfade_shrinking_flush_is_sample_conservative'

Result: 3 passed (targeted overlap/bridge checks).

Manual Smoke Notes

  • Realtime examples were checked against current RealtimeSession API semantics and lifecycle method signatures.
  • Short chunk/overlap decoder smoke showed overlap reduces boundary discontinuity versus no-overlap:
    • overlap=2 boundary jumps: [0.4583, 0.2083]
    • overlap=0 boundary jumps: [0.9583, 0.9583]

Reviewer Guide

Suggested review order:

  1. Runtime tests first (test_moss_tts_realtime_runtime.py, realtime regressions).
  2. inference.py session lifecycle and decoder/backpressure/crossfade paths.
  3. model.py realtime runtime wiring and generation flow.
  4. Realtime examples (moss_tts_realtime_text_deltas.py, moss_tts_realtime_multiturn_agent.py) for user-facing contract parity.

@lucasnewman

Copy link
Copy Markdown
Collaborator

Closing for now. We'll reopen if work on #501 is resumed.

@lucasnewman lucasnewman closed this Mar 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants