Skip to content

Standard dataset config keys are silently ignored by built-in response dataset loaders #3270

Description

@tianyi-zhang-02

Bug

load_response_dataset dispatches with dataset_class(**data_config), and every built-in loader accepts **kwargs — so any standard ResponseDatasetConfig key a loader doesn't support is silently swallowed.

Minimal repro (exactly what the dispatcher does for dataset_name: gsm8k, subset: socratic, split_validation_size: 0.05):

GSM8KDataset(split="train", subset="socratic", split_validation_size=0.05)
# → load_dataset("openai/gsm8k", "main")  — both keys silently dropped
  • GSM8KDataset hardcodes the main config while openai/gsm8k also ships socraticsubset: is ignored with no error.
  • 12 built-in loaders (aime, clevr, dapo_math, deepscaler, gsm8k, helpsteer3, nemogym_dataset, geometry3k, intent, oai_format_dataset, squad, refcoco) never call split_train_validation, so split_validation_size/seed silently do nothing — while the SFT guide documents them as generic dataset keys.

This generalizes the failure mode @RayenTian caught in the #3131 review: "a user who sets subset: all in their YAML gets no error — it's swallowed by **kwargs and the default config loads instead."

Proposed fix

A dispatcher-level warning in load_response_dataset: after resolving the dataset class, warn if a user-set behavioral key (subset, split, split_validation_size, seed) is not accepted anywhere in the class __init__ MRO. MRO-aware because e.g. the intent.py subclasses legitimately consume keys via **kwargs and forward to their base; functools.partial registry entries (AIME variants) are unwrapped. A warning rather than an error, for backward compatibility.

Happy to send the PR (guard + unit tests).

cc @RayenTian @yuki-97

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions