Skip to content

README.md

Latest commit

 

History

History
12 lines (10 loc) · 755 Bytes

README.md

File metadata and controls

12 lines (10 loc) · 755 Bytes

Multi-Task Evaluation Example

Configuring multi_task.yaml

  • eval.defaults defines inference parameters shared by every dataset entry. Override them inside an individual dataset block if needed.
  • eval.datasets enumerates the datasets to evaluate. Each entry should specify:
    • name: a short identifier that appears in logs and dashboards.
    • path: the path to the dataset JSONL file.
    • rm_type: which reward function to use for scoring.
    • n_samples_per_eval_prompt: how many candidate completions to generate per prompt.

IFBench Notes

  • When ifbench is used, slime/rollout/rm_hub/ifbench.py will automatically prepares the scoring environment, so no additional manual setup is required beyond providing the dataset path.