TODO improving nanoGPT training #5
Description
- Look at mixed precision, check lr decay, check the compile arguments (see ENH add MixedPrecision + improve compile #7 )
- AdamW reaching 3.28: Making the code more similar to this version of
modded_nanogpt(@tomMoral )- Get some plot with the number of processed tokens as [this one](https://x.com/kellerjordan0/status/1844820919061287009/photo/1; @tomMoral )
- Try to use the ZeroRedundancyOptimizer (@tomMoral )
- Adding SciOn: we can look at this code (@tonysf, Added scionlight optimizer #6 )
- Adding muOn: we can look at this code (@tonysf )
- Delete the PR Adding Muon #8 which seems in bad state
- Implement SOAOP (@svaiter)