stable-diffusion-2-depth training problems

[Henry-Bi]

Hi everyone, I have a technical question regarding the fine-tuning process of the stable-diffusion-2-depth model, and I'd love to hear your insights.
The model card states:
"This stable-diffusion-2-depth model is resumed from stable-diffusion-2-base (512-base-ema.ckpt) and finetuned for 200k steps. Added an extra input channel to process the (relative) depth prediction produced by MiDaS (dpt_hybrid) which is used as an additional conditioning." This implies that the input channels for self.conv_in layer were increased from 4 (for the latent) to 5 (for the latent + depth map). How can one "resume" fine-tuning from a checkpoint when the model's architecture has been modified?

[asomoza]

Hi, the only people that can do a real resume the training are the ones that did it originally since they have the weights state and the optimizer state, just to clarify, what you're going to do is a finetune of a finetune.

Since the model is already supported by diffusers you just need to adapt the training script you want to use SD2, sadly that model architecture wasn't popular or really used so I don't think we have any training scripts for it.

When you load the model with that repo id the model will be configured accordingly to accept the depth map, you just need to adapt the training script to manage the depth maps for the training.