[Judges] rlhflow pairwise judges #2548
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| for prompt, completion_pair in zip(batch_prompts, batch_completions): | ||
| # Convert prompt to chat format | ||
| instruction = [{"role": "user", "content": prompt}] | ||
| context = self.tokenizer_plain.apply_chat_template(instruction, tokenize=False) |
There was a problem hiding this comment.
I recommend using trl.apply_chat_template from the trl data utils here. We've encountered several issues in the past when applying chat templates to partial sequences, and this approach would be more robust.
While one could argue that we control the chat template in this context, using trl.apply_chat_template ensures that any future modifications to the chat template won't introduce unexpected issues here.
There was a problem hiding this comment.
so both this and the below changes are how the RLHF model recommends to do the scoring... I can check if it works using the chat template
|
|
||
| self.tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True) | ||
| self.tokenizer_plain = AutoTokenizer.from_pretrained(model_name, use_fast=True) | ||
| self.tokenizer_plain.chat_template = "\n{% for message in messages %}{% if loop.index0 % 2 == 0 %}\n\n<turn> user\n {{ message['content'] }}{% endif %}{% endfor %}" |
There was a problem hiding this comment.
Why do you need to override the chat template btw?
|
From their demo code, this is what I get as input for the model: doesn't make much sense to me:
Why not something like this instead: |
|
you are using the instructions from here: https://huggingface.co/RLHFlow/pair-preference-model-LLaMA3-8B right? |
precisely |
What does this PR do?
add support for RLHFlow based pairwise judge