[Bug] The number of declared samples in the recording diverged from the one obtained when loading audio (offset is 0.0)

[johnchienbronci]

@pzelasko

sample rate: 16k
The total length of the sound file: 48.92 s

Read a segment of the audio file:
offset: 0.0
duration: 4.948
and the following error will occur:

[Suppressed AudioLoadingError] Error message: The number of declared samples in the recording diverged from the one obtained when loading audio (offset=0.0, duration=4.948). This could be internal Lhotse's error or a faulty transform implementation. Please report this issue in Lhotse and show the following: diff=-703509, audio.shape=(1, 782677), recording=Recording(id='/shorts-20230702.wav_ln25.0', sources=[AudioSource(type='file', channels=[0], source=shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=[LoudnessNormalization(target=-25.0)])

[extra info] When calling: Recording.load_audio(args=(Recording(id='shorts-20230702.wav_ln-25.0', so
urces=[AudioSource(type='file', channels=[0], source=shorts-20230702.wav')], sampling_rate=16000, n
um_samples=79168, duration=4.948, channel_ids=[0], transforms=[LoudnessNormalization(target=-25.0)]),) kwargs={'channels': 0, 'offset': 0.0, 'duration': 4.948})        
                                                                                                                                
[extra info] When calling: MonoCut.load_audio(args=(MonoCut(id='shorts-20230702.wav-000000-000495_ln-25.0', start=0.0, duration=4.948, channel=0, supervisions=[SupervisionSegment(id='shorts-20230702.wav-000000-000495_ln-25.0', recording_id='shorts-20230702.wav_ln-25.0', start=0, duration=4.948, channel=0, text='xxxxx', language=None, speaker=None, gender=None, custom={'tokens': array([1272, 1032, 1920, 1032,  198,  202,  996, 2002, 2011, 2009, 1060,                                                                                         
       1941, 1908, 1160, 1945, 1675, 1602, 1137,  996,   12,  844, 1003,                                                                            
         18, 1123, 1955, 1545, 1434, 1064,  996, 2003, 2001, 2003, 2004,                                                                            
       1173, 1521, 1412])}, alignment=None)], features=None, recording=Recording(id='shorts-20230702
.wav_ln-25.0', sources=[AudioSource(type='file', channels=[0], source='shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=[LoudnessNormalization(target=-25.0)]), custom={'text': 'xxxxx', 'dataloading_info': {'rank': 0, 'world_size': 1, 'worker_id': None}}),) kwargs={}) 

If the offset is set to 0.1, there will be no error messages.

export LHOTSE_AUDIO_DURATION_MISMATCH_TOLERANCE=0.2, can't not resolve it.

How can I solve this problem?

[pzelasko]

Your manifest seems to be wrong, you say:

The total length of the sound file: 48.92 s

but the Recording manifest inside MonoCut says:

recording=Recording(id='shorts-20230702.wav_ln-25.0', ..., sampling_rate=16000, num_samples=79168, duration=4.948, ...)

The Recording duration should be 48.92 s

[johnchienbronci]

I think MonoCut is correct because I have specified the offset and duration.
kwargs={'channels': 0, 'offset': 0.0, 'duration': 4.948}

audio.shape=(1, 782677) should be changed to (1, 79168)

[pzelasko]

MonoCut seems OK but the Recording inside has incorrect metadata. You can check with lhotse.qa.validate(cut.recording, load_data=True)

[johnchienbronci]

check resut:

I don't understand why the audio offset is always 0.0, and duration is always None, regardless of the specified offset value.

[rank0]: lhotse.audio.utils.AudioLoadingError: The number of declared samples in the recording diverged from the one obtained when loading audio (offset=0.0, duration=None). This could be internal Lhotse's error or a faulty transform implementation. Please report this issue in Lhotse and show the following: diff=-703509, audio.shape=(1, 782677), recording=Recording(id='shorts-20230702.wav', sources=[AudioSource(type='file', channels=[0], source='shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=None)

[rank0]: [extra info] When calling: Recording.load_audio(args=(Recording(id='shorts-20230702.wav', sources=[AudioSource(type='file', channels=[0], source='shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=None),) kwargs={})

[pzelasko]

Set the actual duration and num samples of the audio file in the Recording object. Cut is a slice, so the recording must always have full audio metadata.

[johnchienbronci]

Ok.
but why should full samples be compared with cut samples, cause the error?

diff=-703509, audio.shape=(1, 782677), ...

This problem only occurs when the offset is < 0.00097 and the length of the full audio file is different from the length of the clipped audio file.

it's normal when offset >= 0.00097

[pzelasko]

There’s quick-path code optimization when Cut’s start is 0 and duration == Recording.duration that loads the full audio without slicing. Since your metadata is wrong it errors out. When start is nonzero it applies Cut start and duration and manages to load the slice as expected.

[johnchienbronci]

Ok,
Is this a bug in Lhotse or is it an error due to a problem with the function used?
if function used issue, how can i fix it?

[pzelasko]

It’s not a bug in Lhotse. I don’t know how you created the Recording object but the issue is likely there somewhere.

[johnchienbronci]

I use nemo project to load lhotse dataset (set use_lhotse=True)
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/data/audio_to_text_lhotse.py

[pzelasko]

Can you show the Nemo manifest line for that recording?

[johnchienbronci]

{"audio_filepath": "/a/b/c.wav", "text": "efg", "offset": 0, "duration": 7.755, "sampling_rate": 16000}
{"audio_filepath": "/a/b/c.wav", "text": "abcd", "offset": 10, "duration": 5.0, "sampling_rate": 16000}
{"audio_filepath": "/a/b/d.wav", "text": "abcd", "offset": 0, "duration": 2.0, "sampling_rate": 16000}
....

[pzelasko]

I think we were resolving an issue in NeMo for parsing manifests with offsets a while ago; can you try this with the latest NeMo version (or using the unreleased trunk version)?

[johnchienbronci]

I try NeMo version main or v2.2.0rc1
The same error occurs

File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py", line 216, in run                                                                                        
    self.advance()                                                                                                                                                                            
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py", line 455, in advance                                                                                    
    self.epoch_loop.run(self._data_fetcher)                                                                                                                                                   
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/training_epoch_loop.py", line 150, in run                                                                             
    self.advance(data_fetcher)                                                                                                                                                                
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/training_epoch_loop.py", line 282, in advance                                                                         
    batch, _, __ = next(data_fetcher)                                                                                                                                                         
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fetchers.py", line 134, in __next__                                                                                   
    batch = super().__next__()                                                                                                                                                                
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fetchers.py", line 61, in __next__                                                                                    
    batch = next(self.iterator)                                                                                                                                                               
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/utilities/combined_loader.py", line 341, in __next__                                                                        
    out = next(self._iterator)                                                                                                                                                                
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/utilities/combined_loader.py", line 78, in __next__                                                                         
    out[i] = next(self.iterators[i])                                                                                                                                                          
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 701, in __next__                                                                                        
    data = self._next_data()                                                                                                                                                                  
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1465, in _next_data                                                                                     
    return self._process_data(data)                                                                                                                                                           
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1491, in _process_data                                                                                  
    data.reraise()                                                                                                                                                                            
  File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 715, in reraise                                                                                                        
    raise exception                                                                                                                                                                           
RuntimeError: Caught RuntimeError in DataLoader worker process 0.                                                                                                                             
Original Traceback (most recent call last):                                                                                                                                                   
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 351, in _worker_loop                                                                                 
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]                                                                                                                           
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch                                                                                          
    data = self.dataset[possibly_batched_index]                                                
  File "/opt/NeMo/nemo_lastest/NeMo/nemo/collections/asr/data/audio_to_text_lhotse.py", line 53, in __getitem__                                                                               
    audio, audio_lens, cuts = self.load_audio(cuts)                                            
  File "/usr/local/lib/python3.10/dist-packages/lhotse/dataset/input_strategies.py", line 224, in __call__                                                                                    
    return collate_audio(                      
  File "/usr/local/lib/python3.10/dist-packages/lhotse/dataset/collation.py", line 202, in collate_audio                                                                                      
    audios = torch.stack(audios)                                                               
RuntimeError: stack expects a non-empty TensorList                                             

[extra info] When calling: Recording.load_audio(args=(Recording(id='/corpus/aaaaa.wav_sp0.9_rvb', sources=[AudioSource(type='file', channels=[0], source='/corpus/aaaaa.wav')], sampling_rate=16000, num_samples=47698, duration=2.981125, channel_ids=[0], transforms=[Speed(factor=0.9), ReverbWithImpulseResponse(rir=Recording(id='large-Room079-00039', sources=[AudioSource(type='file', channels=[0], source='/corpus/room/RIRS_NOISES/simulated_rirs/largeroom/Room079/Room079-00039.wav')], sampling_rate=16000, num_samples=32000, duration=2.0, channel_ids=[0], transforms=None), normalize_output=True, early_only=False, rir_channels=[0], rir_generator=None, RIR_SCALING_FACTOR=3.0517578125e-05)]),) kwargs={'channels': 0, 'offset': 0.0, 'duration': 2.981125})                                                           
[extra info] When calling: MonoCut.load_audio(args=(MonoCut(id='/corpus/aaaaa.wav-000000-000268_sp0.9_rvb', start=0.0, duration=2.981125, channel=0, supervisions=[SupervisionSegment(id='/corpus/aaaaa.wav-000000-000268_sp0.9_rvb', recording_id='/corpus/aaaaa.wav_sp0.9_rvb', start=0.0, duration=2.981125, channel=0, text='xxxxxxx', language=None, speaker=None, gender=None, custom={'tokens': array([996,   0])}, alignment=None)], features=None, recording=Recording(id='/corpus/aaaaa.wav_sp0.9_rvb', sources=[AudioSource(type='file', channels=[0], source='/corpus/aaaaa.wav')], sampling_rate=16000, num_samples=47698, duration=2.981125, channel_ids=[0], transforms=[Speed(factor=0.9), ReverbWithImpulseResponse(rir=Recording(id='large-Room079-00039', sources=[AudioSource(type='file', channels=[0], source='/corpus/room/RIRS_NOISES/simulated_rirs/largeroom/Room079/Room079-00039.wav')], sampling_rate=16000, num_samples=32000, duration=2.0, channel_ids=[0], transforms=None), normalize_output=True, early_only=False, rir_channels=[0], rir_generator=None, RIR_SCALING_FACTOR=3.0517578125e-05)]), custom={'text': 'xxxxxxx', 'dataloading_info': {'rank': 0, 'world_size': 1, 'worker_id': None}}),) kwargs={})                                                                                                                         
[Suppressed AudioLoadingError] Error message: The number of declared samples in the recording diverged from the one obtained when loading audio (offset=0.0, duration=4.55225). This could be internal Lhotse's error or a faulty transform implementation. Please report this issue in Lhotse and show the following: diff=-20601636, audio.shape=(1, 20674472), recording=Recording(id='/corpus/aaaaa.wav_sp0.9', sources=[AudioSource(type='file', channels=[0], source='/corpus/aaaaa.wav')], sampling_rate=16000, num_samples=72836, duration=4.55225, channel_ids=[0], transforms=[Speed(factor=0.9)])                                                            
[extra info] When calling: Recording.load_audio(args=(Recording(id='/corpus/aaaaa.wav_sp0.9', sources=[AudioSource(type='file', channels=[0], source='/corpus/aaaaa.wav')], sampling_rate=16000, num_samples=72836, duration=4.55225, channel_ids=[0], transforms=[Speed(factor=0.9)]),) kwargs={'channels': 0, 'offset': 0.0, 'duration': 4.55225})                                                                
[extra info] When calling: MonoCut.load_audio(args=(MonoCut(id='b4862b21-fb97-d435-8856-1712e8e5216a_sp0.9', start=0.0, duration=4.55225, channel=0, supervisions=[SupervisionSegment(id='/corpus/aaaaa.wav-000000-000410_sp0.9', recording_id='/corpus/aaaaa.wav_sp0.9', start=0.0, duration=4.55225, channel=0, text='xxxxxxx', language=None, speaker=None, gender=None, custom={'tokens': array([996,   0])}, alignment=None)], features=None, recording=Recording(id='/corpus/aaaaa.wav_sp0.9', sources=[AudioSource(type='file', channels=[0], source='/corpus/aaaaa.wav')], sampling_rate=16000, num_samples=72836, duration=4.55225, channel_ids=[0], transforms=[Speed(factor=0.9)]), custom={'text': 'xxxxxxx'}),) kwargs={})