MPO

[qgallouedec]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ariG23498]

Here is a rough colab notebook that I have created.

In the notebook:

1. Installed trl from the branch
2. Used the openbmb/RLAIF-V-Dataset dataset
3. Used the HuggingFaceTB/SmolVLM-Instruct model

Here are my queries:

1. Reading the code, we expect the loss_type to be a str with , seperated losses.

   training_args = DPOConfig(
       ...
       loss_type="sigmoid, bco_pair", # <-- a collection of losses
   )

   As far as I understand the MPO paper, it comprises of three losses, the DPO loss, the BCO loss, and the SFT loss. Here I have added the BCO and used sigmoid for the DPO loss, do we also have a way to add the SFT loss here somehow (which I think would be cross entropy loss?)
2. The weighting parameter in the config (as in this comment)

[qgallouedec]

ideally, at this point, we would have

• self.loss_type a list of strings (eg, [“sigmoid”, “bco_pair”])
• self.loss_type_to_weights a dict of str, float which for each loss type associates a weight.

Parsing for loss type could be done directly in the config, as here:

trl/trl/trainer/nash_md_config.py

Lines 34 to 46 in fe4b5ef

	mixture_coef: list[float] = field(
	default_factory=lambda: [0.5],
	metadata={
	"help": "Logit mixture coefficient for the model and reference model. If a list of floats is provided "
	"then the mixture coefficient is selected for each new epoch and the last coefficient is used for the "
	"rest of the epochs."
	},
	)
	def __post_init__(self):
	super().__post_init__()
	if hasattr(self.mixture_coef, "__len__") and len(self.mixture_coef) == 1:
	self.mixture_coef = self.mixture_coef[0]

[ariG23498]

@qgallouedec Here is a rough colab notebook with the current MPO training.

Let me know what you think.

[qgallouedec]

Looks good! Some initial comments :)

[qgallouedec]

I need to check if this work with the parser and the cli

[ariG23498]

If you could let me know how you might check it, I could do it and report back to you.

[qgallouedec]

running this should work:

trl dpo --output_dir tmp_dir --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_preference --report_to none

[ariG23498]

I tried the above command, which resulted in an issue with the self.loss_type being None.

Upon changing the command to

trl dpo --output_dir tmp_dir --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_preference --report_to none --loss-type sigmoid

it started to train. I am not sure why this happens 🤔

[qgallouedec]

Really nice!!!

But don't do this (important):

    # Apply the chat template
    prompt = processor.apply_chat_template(prompt, tokenize=False)
    chosen = processor.apply_chat_template(chosen, tokenize=False)
    rejected = processor.apply_chat_template(rejected, tokenize=False)

The DPO Trainer handle applying the chat template. See #1930 for more info.

This code snippet is present in so many examples online, it's a scourge. 😩

[qgallouedec]

you can use list and dict instead

[qgallouedec]

can you also document that, when a list is passed, the loss is the sum of these values?

[qgallouedec]

add "sft" as well

[qgallouedec]

nice!

[ariG23498]

@qgallouedec I have resolved the merge conflicts and have also worked on the review suggestions. Could you help me with another round of review?

If this looks good, I can start a small training run on VLM with MPO. WDYT?

[ariG23498]

@qgallouedec a gentle ping here!