LLama3 dataset indexing fails

[gyulaz-htec]

I've downloaded the Llama3 dataset with the command from the readme: https://github.com/mlcommons/training/tree/master/large_language_model_pretraining/nemo#dataset.
I'm seeing the following error during build_data_index:

data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/blended_megatron_dataset_builder.py", line 126, in build                                                                 
data_index/0 [default0]:[rank0]:     datasets = self._build_blended_dataset_splits()                                                                                                                      
data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/blended_megatron_dataset_builder.py", line 259, in _build_blended_dataset_splits                                         
data_index/0 [default0]:[rank0]:     blended_datasets[i] = self._build_megatron_dataset_splits(                                                                                                           
data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/blended_megatron_dataset_builder.py", line 417, in _build_megatron_dataset_splits                                        
data_index/0 [default0]:[rank0]:     low_level_dataset = self.cls.build_low_level_dataset(dataset_path, self.config)                                                                                      
data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/gpt_dataset.py", line 150, in build_low_level_dataset                                                                    
data_index/0 [default0]:[rank0]:     return IndexedDataset(dataset_path, multimodal=False, mmap=config.mmap_bin_files)                                                                                    
data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/indexed_dataset.py", line 544, in __init__                                                                               
data_index/0 [default0]:[rank0]:     self.initialize(path_prefix, multimodal, mmap, s3_config)                                                                                                            
data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/indexed_dataset.py", line 584, in initialize                                                                             
data_index/0 [default0]:[rank0]:     self.index = _IndexReader(idx_path, self.multimodal)
data_index/0 [default0]:[rank0]:   File "/opt/megatron-lm/megatron/core/datasets/indexed_dataset.py", line 237, in __init__
data_index/0 [default0]:[rank0]:     with open(idx_path, "rb") as stream:
data_index/0 [default0]:[rank0]: IsADirectoryError: [Errno 21] Is a directory: '/data/data/c4-validation-91205-samples.en_text_document.idx'   

Most probably there is an issue with the folder name like it containing .-s. Could that confuses the file reading?

[gyulaz-htec]

I've managed to get pass the error: moved the two files from c4-validation-91205-samples.en_text_document.bin and c4-validation-91205-samples.en_text_document.idx folders a folder up after that deleted the folders and changed the file names to the previous folder names.

[ShriyaRishab]

What did you rename the files to for it to work?

[gyulaz-htec]

What did you rename the files to for it to work?

I had a file named c4-validationn-91205-samples.en_text_document.idx in the folder and had to rename it to c4-validation-91205-samples.en_text_document.idx. Notice the difference between validationn and validation, the second only have 1 n in it.
The same applies to the other file c4-validationn-91205-samples.en_text_document.bin

[ShriyaRishab]

Oh got it, thanks. @Elnifio can we rename the files?

[ShriyaRishab]

@nathanw-mlc - can we please rename this?

[nathanwasson]

@nathanw-mlc - can we please rename this?

@ShriyaRishab @gyulaz-htec I can rename the files, but I noticed that in addition to there being a typo in the two file names, each of the files is also inside a directory with the same name as the file itself. Is this intended or something I should fix?

The current paths:

mlcommons-training-wg-public/common/datasets/c4/mixtral_8x22b_preprocessed/c4-validation-91205-samples.en_text_document.bin/c4-validationn-91205-samples.en_text_document.bin

mlcommons-training-wg-public/common/datasets/c4/mixtral_8x22b_preprocessed/c4-validation-91205-samples.en_text_document.idx/c4-validationn-91205-samples.en_text_document.idx

I would expect the paths to be like so:

mlcommons-training-wg-public/common/datasets/c4/mixtral_8x22b_preprocessed/c4-validation-91205-samples.en_text_document.bin

mlcommons-training-wg-public/common/datasets/c4/mixtral_8x22b_preprocessed/c4-validation-91205-samples.en_text_document.idx

[ShriyaRishab]

Thanks a lot, Nathan.

I don't think that was intended (even based on the README) so please do rename it.

[ShriyaRishab]

Closing since the original issue if fixed.

[nathanwasson]

I don't think that was intended (even based on the README) so please do rename it.

Okay, I've renamed the files and removed the extraneous directories in the file paths.

[ShriyaRishab]

Great, thanks a lot!