[ASR] Begining of finetuning with lhotse worsen the base model

[AudranBert]

Hi,
I have been trying using lhotse dataloader for ASR for some time. Before that, I was using tarred datasets. I have been trying to finetune this model, everything work great on tarred datasets: it starts at 20% of WER and after some training it goes down to 10%. But, with lhotse it starts at 20% (WER of the model I try to improve) and then a few steps it goes to 80% of WER (10 steps is enough to reach 80%) and after a few thousand steps it starts to go down again.
Is there anybody that faced this issue?

Red curve is with lhotse, blue with tarred datasets. 10 steps.