Set _tokenizer attribute in experimental trainers

[albertvillanova]

Set _tokenizer attribute in experimental trainers.

Follow-up to:

• Set _tokenizer as trainer attribute #5489

---

Note

Medium Risk
Broad but mechanical refactor touching padding/EOS IDs in several training and generation paths; a missed reference could cause runtime errors or subtle masking/padding behavior changes.

Overview
Standardizes tokenizer handling across experimental trainers by storing the resolved tokenizer on self._tokenizer (from either a ProcessorMixin or a PreTrainedTokenizerBase) and ensuring pad_token defaults to eos_token when missing.

Updates all downstream uses (data collators, dataset preprocessing EOS appends, padding/gathering logic, and generation configs/EOS detection) to reference self._tokenizer.* instead of local tokenizer variables or separate self.pad_token_id/self.eos_token_id fields.

^{Reviewed by Cursor Bugbot for commit ba22aad. Bugbot is set up for automated code reviews on this repo. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ad61f84. Configure here.}

[qgallouedec]

lgtm!