Integrate parallel oracle streaming and defer hot-path logging

[yagumana]

Summary

• Integrate parallel oracle LLM streaming into server_oracle.py and keep it as the canonical oracle-guided entrypoint.
• Replace server_oracle_parallel.py with a compatibility shim.
• Guard LLM stream sessions so work from a disconnected session cannot affect a later connection.
• Move high-frequency console and session logging off the asyncio audio path using a bounded background logger.
• Time-slice queued oracle updates to approximately 2 ms per audio-loop iteration so an LLM backlog cannot monopolize real-time audio processing.
• Add tests for parallel generation handling and deferred session logging.

Why

PR #5 introduced parallel LLM streaming as a separate server entrypoint. This PR consolidates that implementation into the canonical oracle server,
avoiding two independent implementations of the same inference path.

Parallel streaming can produce many LLM chunks while the same event-loop thread is responsible for audio processing. Synchronous console and file
logging, combined with draining all queued oracle updates in one pass, could keep the audio loop busy for hundreds of milliseconds and cause audible
interruptions.

Deferred logging removes file and console I/O from the audio path. Time-slicing queued oracle updates ensures that any remaining backlog is handled in
later audio-loop iterations instead of delaying audio processing until the queue is empty.

Behavior and scope

• Preserve the existing ASR and conversation-state semantics.
• Keep opus_loop as the only writer to LMGen; parallel LLM tasks only enqueue oracle updates.
• Preserve existing session-log formats and FIFO ordering for accepted records.
• Use a bounded logging queue; if it is full, log records are dropped rather than blocking inference.
• Leave model weights, sampling settings, ASR configuration, and parallelism limits unchanged.
• Keep internal-only realtime diagnostics and Japanese-specific changes out of this PR.
• Conversational response quality is outside the scope of this PR.

Validation

Static and unit checks:

• Ruff check: passed
• Ruff format check: passed
• Pyright: 0 errors, 0 warnings
• Pytest: 16 passed
• Git diff check: passed

A controlled end-to-end audio-path smoke test used the real KAME/Moshi and Mimi path with deterministic oracle chunks. OpenAI streaming and ASR were
disabled so the test could isolate audio continuity under oracle-update and logging load.

• The client sent 12.00 seconds of synthetic audio as 600 input Opus packets.
• The client received 150 server audio messages and decoded 11.76 seconds of output.
• Steady-state packet gaps had an 80.11 ms median and 80.59 ms p95.
• One 201.3 ms startup/priming gap occurred; no later gap exceeded 100 ms.
• The deferred logger processed 370 records with no drops or write errors.

A manual comparison using the public Hugging Face checkpoint was also performed against the same parallel branch before and after the logging change:

• Audible interruptions were observed before the change and were not reproduced afterward.
• LLM and applied-oracle records remained consistent in both runs.
• Generation-switch frequency was similar before and after the change.

The manual comparison validates runtime continuity and is not intended as a deterministic benchmark of conversational response quality.

[Copilot]

Pull request overview

This PR consolidates the “parallel LLM streaming” oracle server into server_oracle.py, adds session-guarding to prevent stale background work from affecting new connections, and moves hot-path console/session logging off the asyncio audio loop via a bounded background logger.

Changes:

• Integrate a parallel LLM stream multiplexer into server_oracle.py and make server_oracle_parallel.py a compatibility shim.
• Add deferred (threaded) session logging and update the audio loop to time-slice application of queued oracle updates (~2ms budget).
• Add unit tests covering deferred logging and key multiplexer/session-guard behaviors.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_server_oracle_logging.py	Adds tests for deferred session logging integration and multiplexer session/adoption behavior.
tests/test_deferred_logging.py	Adds focused unit tests for DeferredSessionLogger correctness and noop behavior when disabled.
src/kame/server_oracle.py	Becomes the canonical entrypoint; integrates parallel LLM streaming, session guards, deferred logging, and time-sliced oracle event application.
src/kame/server_oracle_parallel.py	Replaced with a thin compatibility entrypoint forwarding to server_oracle.cli.
src/kame/deferred_logging.py	New deferred logger implementation using a bounded queue and background thread.
README.md	Updates documentation to use kame.server_oracle as the entrypoint instead of the former parallel module.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.