SOLVER Implement SOAP

[svaiter]

This PR intends to implement the SOAP algorithm from https://arxiv.org/abs/2409.11321

It is drafted from the original soap.py implementation available at https://github.com/nikhilvyas/SOAP/blob/main/soap.py

OpenAI codex was used to start this feature using the following prompt on the repository:

Adapt original_soap.py optimizer in solvers/soap.py to conform the use of benchopt. Look at adam.py to understand how to modify the code.

Default hyperparameter (lr and weight_decay) were not working on Simulated dataset (3e-3 and 1e-2). Default parameter of Adam neither (1e-3 and 1e-4).
Current working are 1e-4 and 1e-3

TODO

• Review carefuly the generated code.
• Explore hyperparameters.
• Test the DDP on a multi gpu machine

[tomMoral]

seems to be working with only a few tweaks:

But the performances are not super good

[tomMoral]

I did a few extra clean up.
Just one comment on the parameters, it feels like there are too many.

[tomMoral]

A few tweaks that can be helpful

[svaiter]

@tomMoral Shall we merge it like that, or do additional hyperparameter experiences?