grad accumulation steps

[hnyu]

This PR adds a flag for accumulating gradients for optimizer step. It can be used to simulate training with a large batch size with limited memory. The idea is to keep accumulating param grads before triggering a real optimizer step, by scaling the grad with 1/acc_steps.

[emailweixu]

Is it compatible with GradScaler? If not, need to raise error if both are used.

[hnyu]

Is it compatible with GradScaler? If not, need to raise error if both are used.

According to pytorch doc:

 # scaler.step() first unscales the gradients of the optimizer's assigned params.
 # If these gradients do not contain infs or NaNs, optimizer.step() is then called,
 # otherwise, optimizer.step() is skipped.
 scaler.step(optimizer)

So the optimizer already sees grad after unscaling.