Bug
load_response_dataset dispatches with dataset_class(**data_config), and every built-in loader accepts **kwargs — so any standard ResponseDatasetConfig key a loader doesn't support is silently swallowed.
Minimal repro (exactly what the dispatcher does for dataset_name: gsm8k, subset: socratic, split_validation_size: 0.05):
GSM8KDataset(split="train", subset="socratic", split_validation_size=0.05)
# → load_dataset("openai/gsm8k", "main") — both keys silently dropped
GSM8KDataset hardcodes the main config while openai/gsm8k also ships socratic — subset: is ignored with no error.
- 12 built-in loaders (aime, clevr, dapo_math, deepscaler, gsm8k, helpsteer3, nemogym_dataset, geometry3k, intent, oai_format_dataset, squad, refcoco) never call
split_train_validation, so split_validation_size/seed silently do nothing — while the SFT guide documents them as generic dataset keys.
This generalizes the failure mode @RayenTian caught in the #3131 review: "a user who sets subset: all in their YAML gets no error — it's swallowed by **kwargs and the default config loads instead."
Proposed fix
A dispatcher-level warning in load_response_dataset: after resolving the dataset class, warn if a user-set behavioral key (subset, split, split_validation_size, seed) is not accepted anywhere in the class __init__ MRO. MRO-aware because e.g. the intent.py subclasses legitimately consume keys via **kwargs and forward to their base; functools.partial registry entries (AIME variants) are unwrapped. A warning rather than an error, for backward compatibility.
Happy to send the PR (guard + unit tests).
cc @RayenTian @yuki-97
Bug
load_response_datasetdispatches withdataset_class(**data_config), and every built-in loader accepts**kwargs— so any standardResponseDatasetConfigkey a loader doesn't support is silently swallowed.Minimal repro (exactly what the dispatcher does for
dataset_name: gsm8k, subset: socratic, split_validation_size: 0.05):GSM8KDatasethardcodes themainconfig whileopenai/gsm8kalso shipssocratic—subset:is ignored with no error.split_train_validation, sosplit_validation_size/seedsilently do nothing — while the SFT guide documents them as generic dataset keys.This generalizes the failure mode @RayenTian caught in the #3131 review: "a user who sets
subset: allin their YAML gets no error — it's swallowed by**kwargsand the default config loads instead."Proposed fix
A dispatcher-level warning in
load_response_dataset: after resolving the dataset class, warn if a user-set behavioral key (subset,split,split_validation_size,seed) is not accepted anywhere in the class__init__MRO. MRO-aware because e.g. theintent.pysubclasses legitimately consume keys via**kwargsand forward to their base;functools.partialregistry entries (AIME variants) are unwrapped. A warning rather than an error, for backward compatibility.Happy to send the PR (guard + unit tests).
cc @RayenTian @yuki-97