1.0.14 - Adaptive Optimizers and more!

June 3, 2023 Patch Notes (#1251)

• 🎨 New Adaptive optimizers | (see below)
• 📷 Image processing improvements | Added support for transparent datasets images and parsing orientation from exiff
• 🧪 New experimental settings:
• ToMe (Token Merging) - increases training performance at the cost of reduced quality
• Disable Class Matching - Disabled matching rules when collecting the class dataset (useful for pre-made class sets)
• Shared Model Source for LoRA - Enabling will reuse extracted source checkpoints
• TENC controls for weight decay and gradient clip
• 🐛 Bug fixes
• Some new settings

Dadaptation Optimizers!

This release includes adaptive optimizers from Facebook. These new optimizers are much easier to configure and give similar quality. However, they may only work with LoRA on consumer cards due to VRAM limitations.

Recommended settings

The adaptation optimizers have different LR and WD ranges from Torch/8Bit AdamW. Hovering over the optimizers field will show a tooltip with recommendations. Here's an example of a good setup:

✅ Lora
✅ Extended Lora
~80 Epochs (adjust for the amount of training you want)
~0.3 for both Learning Rates
0 Warmup
AdamW Dadaptation Optimizer
❌ EMA
0 TENC ratio
fp16 Mixed Precision
Default Memory attention
0 Weight Decay
Set your Batch Size. 6 is a good starting value. Higher values mean faster training, but it also increases VRAM usage, so you'll OOM if it's too high (also speed will slow down if your VRAM usage is at the limit).

On the Saving tab, set both Lora Rank sliders to 32 (it feels like this quality is a bit better than lower values, I haven't noticed any improvements going above 32)

I'm pretty sure you could use this setup with 8gb. I'm pretty sure this setup is pretty close to optimal for finetuning atm. If you're training a new token, then you might also want to train TENC, which you can do separately if you are low on VRAM. If you need to reduce VRAM usage even further, then you could use xformers, or disable extended lora, or even reduce the resolution slider.

Other features

Portable Concept Lists

Automatically rebuild path from where folder JSON is reside and concat path that defined, allow user share dataset without needing to edit the JSON. No need to use absolute/rooted path on instance_data_dir.

see #1212 for more details.

Other issues

The extension still has some issues with the latest versions of the A1111. If you experience issues, consider using an older commit (such as a9eab236) that uses Gradio 3.16.2.
Some users have also reported issues with Torch 2. We are considering that the "standard" torch version now but consider reverting to Torch 1 if you experience torch issues.