01 Workflow Observation - Accelerating the Learning Rate

[daniel-j-nelson]

While playing with the training loops in 01 Workflow fundamentials I found something interesting that I thought was worth sharing.
An optimizer lr that is very small (0.00001) takes a lot of epocs to lower loss. Also an optimizer lr of 0.1 learns quicker, however can only take the loss/test loss down to a point where it cannot improve no matter how many epocs you give the training loop. Too bad we cant get both...

So I started thinking...what if I change the LR while inside the trainging loop? After the loss hits a certain value (very much related to the lr setting), adjust the lr to a smaller number. I did this is a simple switch statement.

Maybe I am crazy, but this seems to do a pretty good job of reducing epoch count required to get to a small loss as quickly as possible.

match optimizer.param_groups[0]['lr']:
    case  0.1:
        if loss < 0.1:
          print("Reducing Optimzer LR to 0.01")
          parm_group['lr'] = 0.01
    case  0.01:
        if loss < 0.01:
          print("Reducing Optimzer LR to 0.001")
          parm_group['lr'] = 0.001
    case 0.001:
      if loss < 0.001:
        print("Reducing Optimzer LR to 0.0001")
        parm_group['lr'] = 0.0001
    case 0.0001:
      if loss < 0.0001:
        print("Reducing Optimzer LR to 0.00001")
        parm_group['lr'] = 0.00001

[LuluW8071]

@daniel-j-nelson What you are looking for is scheduler which is in built in PyTorch. The scheduler works by comparing previous metrics (say val_loss), if it starts to become stagnant, the scheduler algorithm will decrease the learning rate.

Scroll down to Scheduler

# Decrease LR after each 20 epochs by 0.8% of current LR
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.8) 

for epoch in range(epochs):
    model.train()
   # train loop ...

    model.eval()
    with torch.no_grad():
        # val_loop ....

    scheduler.step()

[daniel-j-nelson]

Thank you @LuluW8071 That's awesome! I figured there had to be a better way than what I was doing.