chore: Allow numpy >= 2.0.0

[booxter]

A large e2e run with forced numpy 2.0+ suggests that the library is compatible with numpy 2.x series.

Also, removed numpy dependency from requirements-dev.txt because it's already listed in requirements.txt.

See: https://github.com/instructlab/training/actions/runs/14539292904

Signed-off-by: Ihar Hrachyshka ihar.hrachyshka@gmail.com

[github-actions]

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

[github-actions]

e2e workflow succeeded on this PR: View run, congrats!

[RobotSail]

@booxter We use numpy in a few critical points like the collator function in the dataloader as well as the data processing logic when converting from messages into raw input IDs for the model to consume. This is a PR where we'd need to run training and validate that the loss curve from your changes produces the same loss curve as what's in the main branch.

[booxter]

Fair enough. We'll need to automate validation for patches like this one (also refactors etc.) I don't feel we should continue with human assessments (or at the very least artifacts - charts etc. - needed for such assessment should be generated and posted in comments.) I will mark this as a draft and report an issue to implement the necessary automation.

[RobotSail]

@booxter Yes, we have a script to do this already but I think it broke and I haven't had time to look into it. If you can fix it then that would be a huge contribution which would automate most of the testing we do manually today. You can find it here: https://github.com/instructlab/training/blob/main/scripts/create-loss-graph.py

And then the actual CI code to call it can be found here:

training/.github/workflows/e2e-nvidia-l40s-x4.yml

Line 387 in 65056fa

python training/scripts/create-loss-graph.py \

[booxter]

@Mergifyio rebase

[mergify]

rebase

✅ Branch has been successfully rebased

[booxter]

Running e2e here again with enforced numpy2: https://github.com/instructlab/training/actions/runs/15446098645

[github-actions]

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

[booxter]

Smoke check here: https://github.com/instructlab/training/actions/runs/15448724050

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @booxter please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[github-actions]

e2e workflow succeeded on this PR: View run, congrats!

[booxter]

Looking at charts, visually, I don't see any difference:

with numpy2: https://github.com/instructlab/training/actions/runs/15446098645
with numpy1: https://github.com/instructlab/training/actions/runs/15443591418

[booxter]

Except visual comparison, is there any additional prerequisite to hit before merging this? @RobotSail

[RobotSail]

@booxter Sorry I didn't see your message prior as I was out on PTO. Ideally we just need to check what APIs we are using from numpy and whether any are impacted. 95% likely it's fine but would be good to double-check.