Request: Enhance audio-text synchronization for RESPONSE_AUDIO_TRANSCRIPT_DELTA and RESPONSE_AUDIO_DELTA events

[opchronatron]

Objective:
Improve the ability to align text and audio deltas for smoother playback and interruption handling.
Proposed solutions (in order of preference):

• Implement corresponding event_ids between text and audio delta events.
• Alternatively, provide approximate audio frame numbers for sentence pauses or completions.

Benefits:
Enables graceful sentence completion before cutting off buffered audio from the previous turn.
Improves overall user experience with more natural speech flow and interruptions.

This would make life much easier :)

[khorwood-openai]

I've forwarded this to the Realtime team! They can sync if there are any updates here.

[Stevenic]

Nice feature request... I suspect it'll be difficult for them todo as it doesn't look like there's a 1:1 mapping of audio to text deltas. They're close but i've seen different lengths...

[mike2505]

Hi, just wondering if there are any updates on this? Happy to help test if needed

[Abdulk084]

hi, just wondering if there is any update or if the team may be working on this? not sure of its possible within the context of realtime model architectures openai has designed?