Manipulating large amounts of Wavs

[domklement]

Hello,

I'm wondering if there's a more efficiet way of handling large datasets in Lhotse similarly to HuggingFace datasets? I can see that datasets consisting of potentially millions of audio files can put strain on the network FS and slow down the data transfer. Also, moving such datasets around can be quite complicated.

I know that Lhotse supports a compressed format shar but it's not suitable for random access during training afaik.

Do you think including support for something like Apache Arrows would be a good idea? This way, one would be able to convert the entire dataset (at least the audio files) into a few shard files and still access the individual files randomly without a significant performance loss.

Thanks for the response.

Best,
Dominik