build(deps): constrain tokenizers below 0.23 so CLIP tokenizers construct

[pruprakash]

Summary

FLUX inference crashes on every rank before generation, in
FluxInferencePipeline.load_text_encoders when constructing the CLIP tokenizer:

TypeError: RobertaProcessing.__new__() got an unexpected keyword argument 'cls'

Root cause: [tool.uv] sets prerelease = "allow" and tokenizers had no upper
bound, so the lock resolved tokenizers 0.23.0rc0 — a release candidate that
renames RobertaProcessing's cls keyword to cls_token. transformers (including
current main) still passes cls=, so any CLIPTokenizer construction raises at
__init__. This blocks FLUX inference and any other path that builds a CLIP tokenizer.

Changes

• pyproject.toml: add tokenizers>=0.22.0,<0.23 to [tool.uv] override-dependencies
  (alongside the existing onnx / langchain pins), resolving tokenizers to 0.22.2.
• uv.lock: regenerated — tokenizers 0.23.0rc0 → 0.22.2.
• tests/unit_tests/utils/test_tokenizers_compat.py: regression guard asserting the
  installed RobertaProcessing accepts the exact kwargs transformers' CLIPTokenizer
  passes (sep, cls, add_prefix_space, trim_offsets) — fails fast if the lock ever
  drifts back to a renaming build.

Why this layer (not the prerelease policy or transformers)

• prerelease = "allow" is global and intentional (other deps here depend on prerelease /
  internal wheels); disabling it would churn unrelated resolutions. Scoping the fix to the
  one offending package via override-dependencies is the surgical, repo-idiomatic move.
• The cap is <0.23, not !=0.23.0rc0: the cls → cls_token rename lives in the whole
  0.23 line, so excluding only the rc would let a future stable 0.23.0 reintroduce the
  crash. Lift the bound once transformers adopts the renamed kwarg.
• transformers is the component behind the API change, but it can't be fixed from this repo;
  pinning tokenizers to the compatible line is the correct lock-side fix.

Verification

• Dep-compat (red-green): RobertaProcessing(sep=…, cls=…, add_prefix_space=…, trim_offsets=…)
  — RED on 0.23.0rc0 (TypeError: unexpected keyword argument 'cls'); GREEN on 0.22.2
  (constructs). This is exactly what the new unit test asserts.
• E2E (nvcr.io/nvidian/nemo:26.06.01.rc0, EOS): FLUX inference — the failing artifact —
  runs the full HF→Megatron convert → inference → denoise pipeline to completion and writes an
  image with tokenizers 0.22.2 (the version this constraint resolves to); it crashed at the
  CLIP tokenizer load before. (The shipped container venv still bakes 0.23.0rc0; the lock fix
  takes effect once a container rebuilds from the updated lock — the e2e confirms 0.22.2 is
  the correct, working target.)

Reference

• NVBug 6317367 (prior conversion-step layer: 6169776).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[claude]

Light review - LGTM

Clean, well-justified pin. The override-dependencies entry tokenizers>=0.22.0,<0.23 correctly excludes the 0.23 line that renamed the RobertaProcessing cls kwarg to cls_token (which transformers still calls with cls=, breaking CLIPTokenizer). The uv.lock update matches (0.23.0rc0 -> 0.22.2), and the comment documents the rationale well.

Minor notes (non-blocking):

• In test_roberta_processing_accepts_transformers_kwargs, the real check is that the constructor does not raise TypeError on the cls= kwarg. The assert processor is not None is trivially true (the constructor never returns None) and adds little; a comment noting that construction itself is the assertion would clarify intent.
• Since this is an upper-bound pin to dodge a breaking rename, consider a TODO/tracking note to revisit once transformers adopts cls_token=, so the <0.23 cap does not silently block future tokenizers upgrades.

Suggested test cases

• tests/unit_tests/utils/test_tokenizers_compat.py::TestTokenizersCompat::test_roberta_processing_accepts_transformers_kwargs

No perf tests impacted.

[pruprakash]

/ok to test 4dac481