Chunking long audio files

[NickDarvey]

I'm wanting to process long audio files* in a memory-constrained environment (i.e. my shitty laptop).

If I understand correctly,

1. the Whisper model was designed to work on 30 second chunks
2. the main loop works on 30 second chunks
3. the previous chunk is used as context, the prompt for the next chunk
4. whisper_context will retain this prompt between calls to whisper_full (unless you pass no_context = true as a parameter)

Does this mean I can just take 30 second chunks of my file and call whisper_full with each chunk, in sequence, and get a similar transcription as I would if I processed it all-at-once?

[1] A ~3 hour audio file meant I needed to allocated ~500MB array of float32 (PCM frames) to pass to whisper_full.

[ggerganov]

Yes, this is correct. As long as you keep the same whisper_context alive, it should work as you expect.

[NickDarvey]

(One addendum: word boundaries need to be considered as shown in the streaming example)

[ggerganov]

Ah yes. I actually didn't take this into account.
If the audio is processed in a single pass, then the decoding part takes care of advancing the window at positions that do not split up the words. However, in your scenario, you will lose that and will suffer from some boundary effects when processing each following chunk.

[NickDarvey]

the decoding part takes care of advancing the window at positions that do not split up the words

Could this functionality be extracted so it can be used for my use case, streaming, and whisper_full_parallel? Something which, when given a chunk of floats, returns any remainder to give you a chance to prepend it to the next chunk.

(I had a look at whisper_decode but I'm not sure how it takes care of advancing the window without splitting up words to be able to answer this myself.)

[NickDarvey]

@ggerganov, I was thinking about adding chunking as a part of a wrapper around my basic .NET bindings.

Do you think the advancing-the-window-without-splitting-words functionality of whisper_decode could be extracted for use by the 'chunker'?

[ggerganov]

It should be possible. Here:

https://github.com/ggerganov/whisper.cpp/blob/1290fc64572f434f2f36721d2e2b0913cec0178a/whisper.cpp#L3631-L3634

Instead of taking all decoded tokens and advancing with the full 30s window, we should keep the existing result_len and seek_delta values in the whisper context and expose them through the API. This way the chunker user code can query these values and know how much to advance the window based on the seek_delta.

It's best to open an issue for this to enhance the stream example.