Realtime

[Bellafc]

Can you make the process (asr+diarization) real time? Many thanks!🙏

[wmantly]

Well over a year old and not even response. I suppose not.

[MahmoudAshraf97]

it's definitely doable, depending on how many resources and efforts you are willing to pour into, 'realtime' is a very vague word when it comes to speed because it cannot be measured or defined as a metric, hence the lack of response

[wmantly]

Reasonable. lets refine "real time" by a the standards of a realistic use case: whisper-diarization is listening in on some multi-person audio call and generating a best effort "who said what" as the audio stream in or chunks roll in. Now, its understandable that the STT wont be instant. The "real time" request would be whisper-diarization take an audio stream, or some set audio chunk size in a loop and spits out data as soon as it can.

It is very reasonable to divert the actually audio chunking, chunk queuing and feeding to be an external issue and not whisper-diarization responsibility. But, the chunks should be treated as an ongoing session.

Does this help?

[MahmoudAshraf97]

from the perspective of performance constraints, it's doable since everything has rtf > 1x, what needs work is the individual components that are designed to work with the whole audio available such as STT and diarization model (#342 implements a model with streaming support), ASR and diarization are two independent components and each of them can be adapted to streaming, you'll have to modify the code that combines the results of both components to generate a diarized transcript.

Whisper works the best with 20-30s segments, so for streaming it needs special finetuning such as This, or use a model with native streaming support.
As for diarization, it needs to maintain a session state so that the incoming audio is cross-referenced with the old audio, this is mandatory for it to work