Skip to content

fix(data): warn when behavioral dataset config keys are silently ignored#3271

Open
tianyi-zhang-02 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
tianyi-zhang-02:fix/warn-ignored-dataset-config-keys
Open

fix(data): warn when behavioral dataset config keys are silently ignored#3271
tianyi-zhang-02 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
tianyi-zhang-02:fix/warn-ignored-dataset-config-keys

Conversation

@tianyi-zhang-02

@tianyi-zhang-02 tianyi-zhang-02 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

load_response_dataset instantiates with dataset_class(**data_config), and every built-in loader accepts **kwargs — so a standard ResponseDatasetConfig key the class doesn't support is silently swallowed. E.g. subset: socratic on gsm8k silently loads the hardcoded main config, and split_validation_size/seed silently do nothing on the 12 loaders that never call split_train_validation (while the SFT guide documents them as generic keys).

This adds a dispatcher-level guard, warn_on_unsupported_dataset_config_keys, that warns when a user-set behavioral key (subset, split, split_validation_size, seed) is not declared anywhere in the resolved class's __init__ MRO.

Design notes:

  • MRO-aware: kwargs-forwarding subclasses that delegate to a declaring base (the intent datasets) are not flagged.
  • functools.partial registry entries (AIME variants) are unwrapped before inspection.
  • None values and falsy split_validation_size are skipped, so documented config boilerplate (subset: null, split_validation_size: 0) stays quiet — none of the shipped examples/configs emit warnings.
  • Warning, not error, for backward compatibility.
  • Known limitation: an external dotted-path dataset class that consumes one of these keys via kwargs.get(...) without declaring it in any __init__ signature would see a spurious warning. None of the built-in loaders do this (verified), and it's warning-only by design — but flagging it for review. Scoped to the response dispatcher; the preference dispatcher has the same pattern and can adopt the shared helper as a follow-up.

Issues

Closes #3270. Generalizes the failure mode @RayenTian flagged in the #3131 review.

Usage

data:
  train:
    dataset_name: gsm8k
    subset: socratic   # before: silently loads "main"; after: UserWarning at dispatch

Before your PR is "Ready for review"

  • New tests: 7 unit cases for the guard (unsupported key warns; supported key quiet; None/0 quiet; MRO forwarding quiet; partial unwrapped; non-class input safe) plus an end-to-end dispatcher test, in tests/unit/data/datasets/test_response_dataset_registry.py (registry-dispatch scope).
  • Validation: helper cases verified locally (all pass); ruff check / ruff format --check clean; pyrefly==0.24.2 (CI-pinned) reports no new errors on the two whitelisted files touched.
  • Docs: N/A (warning makes existing documented behavior observable; no doc changes required).

cc @RayenTian @yuki-97

load_response_dataset instantiates with dataset_class(**data_config) and
every built-in loader accepts **kwargs, so a standard ResponseDatasetConfig
key the class does not support is swallowed with no feedback — e.g.
subset: socratic on gsm8k silently loads the hardcoded 'main' config, and
split_validation_size/seed silently do nothing on the 12 loaders that never
call split_train_validation.

Add a dispatcher-level guard that warns when a user-set behavioral key
(subset, split, split_validation_size, seed) is not declared anywhere in
the resolved class's __init__ MRO. MRO-aware so kwargs-forwarding
subclasses (intent datasets) are not flagged; functools.partial registry
entries (AIME variants) are unwrapped; None values and a falsy
split_validation_size are skipped to keep documented config boilerplate
quiet.

Closes NVIDIA-NeMo#3270

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>
@tianyi-zhang-02
tianyi-zhang-02 requested review from a team as code owners July 17, 2026 20:00
@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard dataset config keys are silently ignored by built-in response dataset loaders

2 participants