ENH: Avoid unreported deletion of spectrograms during the training process

[dovi-yellin]

Currently, if the total duration specified by 'train_dur', 'val_dur', and 'test_dur' is shorter than the combined duration of all wav-based spectrogram files, any excess files generated during the prep stage are automatically deleted during training. In this case, the current process also does not report on the selection of files to be deleted.
It would be helpful to offer an option to retain the excess files in a separate folder, or at least provide a log reporting which files were deleted.

[NickleDave]

Thank you @dovi-yellin for raising this issue.

The logic that removes files not assigned to a split of the dataset was added in
cbc3f82

The reasoning is here on the pull request #718 that fixes issue #717.

Practically speaking, the reason I made this change (as explained on the PR + issue) is that we prep datasets we use for running tests in continuous integration. I realized that I was moving gigantic tar.gz files around full of data we weren't actually using. In the worst case, this uses up all the space on GitHub Actions and causes the CI to fail.

By extension: I think the default for vak should be to minimize the amount of data generated. By the principle of least surprise, as a user, I don't want to ask vak to prep a dataset with small split durations, and then find out that it filled my entire hard drive with spectrogram files.

I.e., I think the fact that vak currently computes spectrograms for all audio files, and then deletes some, is a design flaw that should be fixed. I think there are historical reasons that we do this. My guess is that we want to know exactly how many time bins / frames there are in all possible spectrograms when we assign an (audio, spectrogram) pair to one of the splits. But we can do this without actually computing the spectrogram.

It would be helpful to offer an option to retain the excess files in a separate folder, or at least provide a log reporting which files were deleted.

Because I think we should fix prep so that it doesn't generate all those files, I am mildly 👎 on adding an option to keep them.

I do think it should be possible for you to prep however many files you want, and then give them to vak in whatever way you want to, with minimal work. I think we have this functionality, but it's not well documented.

Can you say more about why you would want all those excess files? Is your goal is to generate all possible spectrograms and then make many different permutations of splits from those? If you want to do that, and then tell vak which split to use through a config file, it's definitely possible. You need to make sure you have a csv file representing each split, in the same format that vak uses. You would then point to that file with the splits_path key-value pair in the [vak.train.dataset] table of the config file. Like this:

[vak.train]
standardize_frames = false
batch_size = 11
num_epochs = 2
val_step = 50
ckpt_step = 200
patience = 4
num_workers = 16

root_results_dir = "./tests/data_for_tests/generated/results/train/spect_mat_annot_yarden/TweetyNet"

[vak.train.dataset]
splits_path = "./path/to/splits/you/made/not/using/the/cli"
params = { window_size = 88 }

[vak.train.model.TweetyNet.network]
conv1_filters = 8
conv1_kernel_size = [3, 3]
conv2_filters = 16
conv2_kernel_size = [5, 5]
pool1_size = [4, 1]
pool1_stride = [4, 1]
pool2_size = [4, 1]
pool2_stride = [4, 1]
hidden_size = 32

[vak.train.model.TweetyNet.optimizer]
lr = 0.001

[vak.train.trainer]
accelerator = "gpu"
devices = [0]

Please let me know if that all makes sense. Happy to talk through this more.