Skip to content

Parallelizes LLM streaming - #5

Merged
Kuroki1931 merged 7 commits into
mainfrom
improve_inference
May 13, 2026
Merged

Kuroki1931 merged 7 commits into
mainfrom
improve_inference

Conversation

@Kuroki1931

@Kuroki1931 Kuroki1931 commented May 12, 2026 •

Copy link
Copy Markdown
Collaborator
  • Parallelizes LLM streaming requests during user speech.
  • Starts new LLM generations from updated ASR context while keeping older in-flight generations
    alive briefly.
  • Adopts the newest generation once it begins emitting tokens.
  • Resets the oracle stream when switching to a newer generation.
  • Drops stale generations so only the latest returned stream is fed to Moshi.
  • Keeps Moshi/audio generation on the existing single-writer path to avoid concurrent LMGen access.

@Kuroki1931 Kuroki1931 changed the title parallle initial commit Parallelizes LLM streaming May 12, 2026
@Kuroki1931
Kuroki1931 marked this pull request as ready for review May 12, 2026 23:58
@Kuroki1931
Kuroki1931 requested a review from yagumana May 12, 2026 23:58
continue

# forward only if this gen is the adopted speaker
if gen_id != self.adopted_gen:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this guard should reject older generations as well:

if gen_id <= self.adopted_gen:
    return

The caller checks gen_id > self.adopted_gen before calling _adopt_generation(), but that check happens outside _adopt_lock. If a newer generation is adopted while an older generation is waiting to enter this lock, the current equality-only check would still allow the older generation to set adopted_gen, causing adopted_gen to move backwards.

Since adopted_gen should be monotonic, this should be rechecked inside the lock.

@Kuroki1931

Copy link
Copy Markdown
Collaborator Author

Since this is a quick fix for the surface-level inference code, we will merge it for now. We will improve it later by integrating it with server_oracle.py.

@Kuroki1931
Kuroki1931 merged commit 2fd189c into main May 13, 2026
2 checks passed
@Kuroki1931
Kuroki1931 deleted the improve_inference branch May 13, 2026 04:01
@Kuroki1931

Copy link
Copy Markdown
Collaborator Author

This PR is for #4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants