pretrained model which can resume training

[emmacirl]

@rafaelvalle Thanks for your sharing! It helps a lot.
I find it trains very slow (about 1 epoch/day,batch size=1) when I trained model using my data(about 12h). Could you offer a model which can resume training from it ?

Thanks a lot !!

[jiqizaisikao]

Hi,it seems that it is too much slow,how many epochs should be done before we can get good result

[candlewill]

I use the following method to retrain on the official released pre-trained model:

1. change config.json:
   set "checkpoint_path": "models/waveglow_new.pt"

2. change train.py:

def load_checkpoint(checkpoint_path, model, optimizer):
    assert os.path.isfile(checkpoint_path)
    checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
    # iteration = checkpoint_dict['iteration']
    iteration = 1
    # optimizer.load_state_dict(checkpoint_dict['optimizer'])
    model_for_loading = checkpoint_dict['model']
    model.load_state_dict(model_for_loading.state_dict())
    print("Loaded checkpoint '{}' (iteration {})" .format(
          checkpoint_path, iteration))
    return model, optimizer, iteration

The training is still in process, and I can't tell if this is completely right.

[belevtsoff]

@candlewill I'm trying to finetune the official pretrained model and I'm getting nan's because the determinants of the Intertible1x1Conv weight matrices are negative, so the logdet operation produces nan's. Have you changed something to address that? @rafaelvalle what do you think about it? This apparently doesn't matter for inference, but I'm thinking that you've probably used a slightly different version of code to train that checkpoint (i.e. using abs of the det or smth).

[WendongGan]

@candlewill Thanks for your share! I want to know how about your retraining？ Does it work well?

[WendongGan]

@belevtsoff @candlewill @rafaelvalle I also meet the problem when I try to finetune the official pretrained model.

Look forward your help!

[belevtsoff]

@UESTCgan it seems like the problem is twofold:

1. the glow_old.py uses interchanging splitting of channels (in both forward and infer), whereas glow.py always uses first half of channels as audio_0. So it seems that right now to finetune the thing you have to either modify glow.py to use the old style channels or uncomment and fix the forward method in glow_old.py and use that one instead.
2. in glow.py, replace torch.logdet(W) with torch.det(W).abs().log(). This makes more sense mathematically and will get rid of nan's

I'm not sure if I've spotted everything needed though, but at least now my finetuned model produces speech and not just a bunch of noise

[WendongGan]

@belevtsoff Thanks for your reply! I will have a try, and then I will share my results to communicate with you. Thanks again!

[rafaelvalle]

@belevtsoff careful with the alterations. We initialize the determinants to be positive and the determinant's crossing between positive and negative values suggests that during optimization one is stepping over infinite error, at determinant 0, which is bad. This can be caused by a large update caused by either large learning rate or some outlier batch...

[belevtsoff]

@rafaelvalle Thanks for the response! But actually my point is that if you simply take the matrix W of the InvConv layer from the official pre-trained checkpoint (without any finetuning) - it's determinant will be negative. How is it possible?

[rafaelvalle]

Probably because in the old model we did not enforce the determinant to be 1 at initialization.

[belevtsoff]

@rafaelvalle yeah, probably

Anyway, I've successfully finetuned the pretrained checkpoint using the recipe above. Although it didn't completely remove that reverb-like effect on a male voice.

[candlewill]

After two weeks' fine tune, I can also get some clear voice. However, the problem as @belevtsoff mentioned, also occurs in my experiment. Maybe more training is needed.

Here are some samples (In Chinese):

nvidia_waveglow_samples.zip

[hdmjdp]

@candlewill your wav is female or male?

[candlewill]

@hdmjdp Female.