How to limit the length when merging audios to 30 seconds

[rose-jinyang]

Hello @kenfus and @ClaudioPaonessa
How are you?
Thanks for contributing to this project.
I can not find how to limit the length when merging audios to 30 seconds.

[ClaudioPaonessa]

I am not sure if I understand the request correctly. Do you mean the constant called DURATION within data_processor.py?

[kenfus]

Ah, I think she means that from the start the length of a generated SRT file is 30 seconds.

whisper-prep/src/whisper_prep/generation/generate.py

Line 179 in 3fd24d5

n_samples_per_srt: int,

We have this one, it decides how many samples are used to generate one SRT File but it's not by audio duration.

[rose-jinyang]

Hi
Thanks for your quick replies.
I know that the limits of both audio's entire length and each segment's length for timestamp are 30s when training an Whisper model.
But I can not see where the logic is implemented at.

[kenfus]

whisper-prep/src/whisper_prep/generation/data_processor.py

Line 352 in 3fd24d5

def _create_records_with_timestamps(

This is here. Basically, it iterates trough the audio via the subtitles and accumulates them until they are longer than 30 seconds. Then, it creates one training example for whisper (with language tag, prompt etc).

[rose-jinyang]

Thanks for your explanation.
In fact, I am going to apply your method to Indic speech recognition.
I have another problem.
Mostly, for Telugu (one of Indian languages), when tokenize a transcription-text for audio segment of which length is 30s, length of tokens exceeds 448-token limit (Whisper's decoder token capacity).

In fact, when using sentence-level training data, this problem exists but not occurs occasionally.
But when using your long-form generation method, most of audio segments are 30s in length.
How can we fix this problem up?

[kenfus]

Hm.. How many are filtered out? All of them?

Can you give me some of your data examples but looking at the code, we could probably change this here:

            if tokens_length > self.max_tokens_length:
                tqdm.write(
                    f"Skipping {audio_path} ({format_timestamp(segment_start / 1000)}-"
                    f"{format_timestamp(segment_end / 1000)}) because it is too long "
                    f"({tokens_length} tokens)"
                )
            else:
                record = Record(
                    audio_path=segment_audio_path,
                    language=self.language,
                    text="".join(segment_text),
                    prompt=prompt,
                )
                records.append(record)

The prompt length is shortened anyway in the whisper-finetuning code, if it's too long as input for training, so we could add the same logic here. However, I don't quite understand why it works fine for German but not for your case. Is it tokenized differently? Perhaps the language is not fully supported by Whisper / the tokenizer?