This is great, and I agree that this is the natural next step after merging #104.
To simplify our approach without the burden of maintaining extensive configuration files, have you considered adopting the Croissant metadata format? Croissant is a high-level format for machine learning datasets that combines metadata, resource file descriptions, data structure, and default ML semantics into a single file. It's designed to enhance dataset discoverability and interoperability across ML frameworks. HuggingFace Datasets supports Croissant, see for instance here: https://huggingface.co/datasets/ServiceNow/CORD-1k-Docintel/blob/main/README.md. Croissant could streamline our dataset management and align with existing tools. Could you explore if Croissant would fit our needs here?

That's a promising idea, though it's a lot more involved than what I'm trying to achieve here, and I'm not too familiar with Croissant (it would be a good opportunity to involve other team members). An important question is whether we would like to adopt it before and/or after dataset preparation. How about we discuss it in our next team meeting?

In the meantime I'll prepare a quick POC (solution 3) so we have something to work with.

To eliminate or at least reduce the risk of folder changes going unnoticed, the directory parsing process can generate a list not just in memory but also write it to a file, for example. Alternatively, another type of signature can be created for the folder. When the folder is scanned a second time, an exception is raised if the files differ. Only after examination and approval will it allow the creation of the dataset.

@jlamypoirier, which PRs need to be merged first before @bigximik can work on this?

For the implementation, it goes after #104 and #121 (only need to get the new tests to pass), and I'd like to do the POC I mentioned above first. But it should already be possible to plan and start, especially since much of the work will be in the preparator which there PRs don't really touch.

Also this issue is strictly about auto concatenation, I'll make a separate one for metadata.