Support Python 3.12 for full training

[booxter]

There are concerns about dolomite not supporting Python 3.12+. And indeed, attempts to run large e2e in instructlab/instructlab#3101 with dolomite on failed so far (though perhaps not in training; the last failures I saw were during sdg phase - so we'll have to explore specifically if anything in training is broken).

Regardless, to claim support for python 3.12 for the library, we should

• enable unit tests for it (already done);
• enable smoke tests for it (ci: add 3.12 smoke workflow flavor #535);
• make sure that e2e run in ilab - or an equivalent training specific job - successfully completes with acceptable quality (loss etc.); - ci: Introduce python 3.12 e2e large job flavor #563
• docs updated, if needed;
• package metadata updated, if needed.